AutomaticBugReporter

<<TableOfContents: execution failed [Argument "maxdepth" must be an integer value, not "[2]"] (see also the log)>>

Proposal

Automatic error detecting and reporting has become common practice in desktop and web-based applications. The bugs found in the field may be types that wouldn't ever be caught during testing. With an automatic bug report system, filing a bug report is just a click on a button in a dialog. Many more bug-reports from various kinds of users become possible, including non-technical users and those who can't be bothered to login to a bug tracker and file a detailed report.

Cytoscape should consider a similar automatic bug reporting system, which could potentially increase feedback from Cytoscape users and provide developers useful information needed to fix the bugs. This RFC describes how we might build an automatic bug reporter for Cytoscape.

Use Cases

1. GUI mode

Scenario: A user is using Cytoscape and suddenly Cytoscape runs into a problem. An exception is caught and the exception is classified as a bug. A dialog box pops up and asks user if he/she wants to report the error to Cytoscape. If user says “Yes”, information about the bug will be sent to Cytoscape bug tracker.

2. Headless mode

• Interactive mode: If Cytoscape is running in headless mode and the mode is “interactive”, then when an exception is caught and the exception is classified as a bug, a message on the console will tell the user what happened and ask the user if he/she wants to report the error to Cytoscape. If the user says “Yes”, information about the bug will be sent to Cytoscape bug tracker.

• Server mode: Cytoscape is running in headless mode and the mode is “server”. Cytoscape will check the property “AutomaticBugReporter_alwaysReport”. Report the error only if this property value is true.

3. Do not report any bug

In the case that a user decides not to report any error, the user has the option to set the property value “automaticErrorReport” (true, by default) to false and Cytoscape will remember the decision.

Collecting data from Cytoscape user

When Cytoscape crashes or exception thrown, the following information from user should be collected and submitted directly to Cytoscape bug tracker, Mantis

• Cytoscape version
• OS version and other system info
• The class and line number in the code where the crash occurred
• The error message (full stack trace)
• User’s Description (Optional)
• User email address (Optional)
• Command history – everything the user did since startup – recorded by Cystocape logger
• Stack dump from OSGi that tells us what bundles are loaded, versions, etc

The category of bug report in Mantis may be infered from the specific exception type. We encourage users to give a description of what they did before Cytoscape ran into the problem. This will be very helpful for developers to reproduce the bug. User email address is optional. Many users may not give us their email address at all. However, in cases where the same bug is reported multiple times, and developer has questions about it, it could be useful to contact user if he/she did provide an email address

Format of bug report and identification of duplicated bugs

The subject (summary) of bug report should have the format something like this:

ExceptionType:ClassName:Method:LineNumber:OS:OSVersion This string is used to determine if a bug has already been reported.

If a bug has already been reported (with the same subject), the report should be appended to the existing one, rather than opened as a new one.

Mantis bug Data structure:

OS: Set (WIN, LINUX, MAC, OTHER)

OS version: XP/???

Product build: Cytoscape 3.0

Description:StackTrace

Additional Information: User’s comment, User’s e-mail contact, if any

Steps to reproduce:Command history

Attached file: If session file is attached, it will be added as an attached file.

Notes:If duplicated bug, add a note only,including OS, OS_Version, User’s e-mail,User’s comment, Time stamp

Automatic bug reporter could be part of CyLogger

Since Cytoscape 2.6.1, Cytoscape has its own Logger, CyLogger, which catches all kinds of error messages and present them to the users or developers. CyLogger may be extended to include the automatic error reporter.

The primary job of the reporter is to report RuntimeException, such as NullPointerException, ArrayOutOfBoundException, etc. The sources of the exception may be identified through different types of exception, for example, CyModelException, CyViewException, CyIOException, CyVSException. In turn, the type of CyException can be used to categorize the bug report. In this way, we don’t need to set ErrorCode for each report.

Implementation

The implementation of the automatic bug reporter includes two parts, client side (Cytosape) and server side (Mantis). The client would be responsible for data collecting and submission and the server for data processing. If a task can be done in either client or server side, we prefer to put it on server side. This is important for easy maintenance in the future.

Cytoscape (client side)

Two Java classes in cytoscape.logger package are needed for automatic bug reporter.

1. BugReport.java: Java bean class, to hold bug data

2. BugReportUtil.java: This class is responsible for the bug data validation and submission.

Mantis (Server side)

1. Virtual person: Automatic bug reporter should have its own user name “Automatic bug reporter” in Mantis -- a virtual person, instead of a real developer.

2. Mantis PHP script (bug_report_auto.php): the new Mantis PHP script will be triggered by “BugReportUtil.SubmitBugReport(URL, BugReport)” of Cylogger. This PHP script will (1) check the summary (subject line of the bug report) of the Bug report (2) if this is a new bug, create a new Bug in Mantis, if this is a duplicated bug, append the new report to existing one.

Web page about fixed-bugs

We may need a web page to list bugs-fixed (CyBugsFixed.php) at Cytoscape web site. On the page, fixed bugs for each Cytoscape versions would be listed, so that users could search for the bugs they care about and get advice/suggestions on possible work-arounds. Currently users can search for particular bugs in Cytoscape bug tracker, Mantis. But this is mainly for developers, not for ordinary users.

Integrating Mantis and Subversion

Mantis can be integrated with Subversion, so that if a developer fixes a bug and checks in the change into Subversion, the change can be immediately reflected in Mantis. There is a step-by-step instruction on how to setup the post-commit script in Subversion to make this happen, here.

Related RFCs

Link to other related RFCs

Issues

List any issues, conflict, or dependencies raised by this proposal

Comments

MikeSmoot

• How would ErrorCodes get assigned? Would we need to manually assigned a code to every exception that gets thrown? Would it be easier to assign codes based on the Exception or the package that contains the Exception?

• This seems to dovetail with merging the error messages produced by CyLogger and those produced by the Task framework.

• I would expect the bug report to include the version of cytoscape and possibly a status dump from OSGi that tells us what bundles are loaded, versions, etc..
• It would be useful if the dialog provided an option for users to submit their session file along with the report.

• It would be really useful if we had a command provenance or history system that could tell us everything the user did since startup and we were able to include that in the report.

Peng-Liang Wang

• Agree, it is easy to assign ErrorCode based on type of exception.

• Version of Cytoscape and OSGi status dump are included in BugReport

• Session file should be very useful, but it could be very large.
• If we have command history, it will be included in bug report field -- steps to reproduce

PietMolenaar

• I assume plugins can use the same mechanism, but how should this be implemented at the server side? Should that be something configurable to your own needs? Or direct bugs to specific parts of Mantis?

PengLiangWang

• Yes, Plugin can use the same mechanism to report bug to plugin's own bug tracker. In the case of Mantis, at the server side, we can use either the webservice API (MantisConnect) or Mantis core API to create a customized PHP script.

GaryBader

• Great idea! This may overload Mantis, since there could be thousands of bugs reported about the same issue - would these all be reported under the same bug ID (i.e. avoid duplication)? Additional information could be added as a note.
• There should be an option for automatic reporting of bugs without asking the user. This could be set as a property and would be useful for headless mode running as a server.
• There should be a way to turn this off, with a property, for when people are debugging code.
• Would this report plugin bugs too?
• Would people be able to change the Mantis server that bugs are reported to? That could be useful when you are debugging in a local user environment.

Peng-Liang Wang

• If a bug is determined “duplicated” of existing one based on its summary (title), new issue will not be created in Mantis, instead it will edit the existing one, add a note for new information.

• We can add a property, say “AutomaticBugReport_always = true/false”, to support the headless “server” mode.

• User has the option to turn it off by setting the property “AutomaticBugReport= true/false”

• Core plugin should report bugs to Cytoscape, other plugins should report to their own bug tracker if they wish.

• I think the URL to report Cytoscape bug should be something like this: http://cytoscape.org/automaticBugReport.php Any request to this URL is then redirected to the Mantis server. The reason is because we may change the location of Mantis server, or even replace Mantis with other bug tracker.

BrianTurner

• Bug reporting should really be linked to an overall exception handling strategy. In this article there is an interesting idea about how to avoid logging exceptions multiple times and uniquely identifying an exception--the article's focus is on EJB, but I think some of the ideas still apply in particular:

public class LoggableEJBException extends EJBException {
    protected boolean isLogged;
    protected String uniqueID;
    public LoggableEJBException(Exception exc) {
        super(exc);
        isLogged = false;
        uniqueID = ExceptionIDGenerator.getExceptionID();
    }
        ..
        ..
}

try {
    OrderHome orderHome = EJBHomeFactory.getInstance().getOrderHome();
    Order order = orderHome.findByPrimaryKey(Integer id);
} catch (RemoteException re) {
    LoggableEJBException le =
       ExceptionLogUtil.createLoggableEJBException(re);
    String traceStr = StackTraceUtil.getStackTrace(le);
    Category.getInstance(getClass().getName()).error(le.getUniqueID() +
":" + traceStr);
    le.setLogged(true);
    throw le;
}

Peng-Liang Wang

• Agree. Bug reporting should be linked to exception handling strategy. We need to define different types of CyException.

DanielAbel

• The framework should make it possible to submit a report even if no exception happened. This would allow using the "collect version, debugging information and session file and submit it somewhere" aspect for problems that aren't manifested as exceptions. For example, rendering bugs, or in general buggy behavior that doesn't result in a crash.
• It would be nice if the framework could make it possible to append to a bug (instead of creating a new one). This way one could reproduce the bug on a different platform / OS and append the crash report to the original bug, or ask the bug reporter to append a crash report to a bug report that was created without one.

Peng-Liang Wang

• Yes, not all bugs will crash Cytoscape. A bug might be an error – Cytoscape may produce in-correct results under certain circumstances. In this case, we may need a menu item, say Help --> Report a bug/issue to Cytoscape. A click on it will pop up a Bug Report Form. This will give user a chance to report a bug on Cytoscape rather than login to Cytoscape bug tracker and report it, which is much easier and conveniance for user.