App Store Code Structure

Terminology

• Django app: Django organizes websites into separate modules called apps. Each app has its own directory at the top level typically containing files like __init__.py, model.py, and views.py.

• templates: HTML files with placeholders, which Django processes by filling in Python code.

• static files: general website files (images, Java Script, CSS) that are served as is without any processing from Django.

• media files: general website files referenced by the database; they are served as is without any processing from Django.

• mod_wsgi: Apache module that interfaces with Python.

Explanation of important files

Configuration Files

• settings.py: Django settings file for configuring things like the database, location of templates, static files, and so on.

• urls.py: the general URL layout of the entire site. Each URL entry in this file delegates URL paths to each Django app.

• django.wsgi: the configuration file used when the App Store is deployed to an Apache server using mod_wsgi.

Django Apps

• apps: navigation of Cytoscape apps and app pages

• users: user login/logout

• search: free text searching

• backend: JSON representation of 3.0 apps; used by the App Manager in Cytoscape 3.0+

• help: about, contact us, getting started pages

• submit_app: Cytoscape 3.0 app submission pages and jar verification

• download: for downloading releases and tracks download stats for apps

Other Directories

• templates: Templates used throughout the App Store.

• static: Each subdirectory has static files for a Django app. The common subdirectory has static files that belong to the entire site. When deploying the site to Apache, Apache should directly serve these files instead of through Django.

• util: small utility functions used throughout the site's code

• dbmigration: scripts that directly update SQL tables after changes had been made to database models; only needed when needing to migrate old versions of SQL database backups

• conf: individual configuration files

App Store Software Dependencies

The App Store requires the following software packages. If you're on a Mac, this can be installed with Homebrew. If you're on Linux, use your distribution's package manager.

Note that the versions specified here are not mandatory. They only indicate the version with which I've tested the App Store.

• Python 2.6
• xapian 1.2.13 (free-text searching)
• xapian-bindings 1.2.13

  • This package is not available through Homebrew. The package must be downloaded and installed manually. When running the configure script, make sure to add the --with-python argument: ./configure --with-python

• libjpeg 8d (used by PIL)
• libpng 1.5.14 (also used by PIL)
• GeoIP 1.4.8 (converts IP addresses to geographical locations)

The following Python packages are also required. Each can be installed with pip install. If you don't have pip, type: easy_install pip.

• Django 1.4.5
• MySQL-Python (aka MySQLdb; can be installed with Debian/Ubuntu's package manager; Django uses this to connect to the MySQL database)
• PIL 1.1.7 (can be installed with Debian/Ubuntu's package manager; needed to scale icon and screenshot image files)

  • PIL must be built the JPEG support. At the end of PIL's installation, you'll see a printout titled PIL 1.1.7 SETUP SUMMARY. This must list JPEG as supported.

• django-social-auth 0.7.23 (aka social_auth; allows users to log in with their Google accounts)
• IPython -- optional (can be installed with Debian/Ubuntu's package manager; very useful for debugging)

Testing App Store Software Dependencies

Run the test_dependencies.py script like so:

• python external_scripts/test_dependencies.py

Test the GeoIP library:

• python manage.py test_geoip

Note: this script can only be run after the Django project has been configured.

How requests are handled

         ||             |                               |                    |                    |            ||
Request=>|| ==Apache==> | == sites-enabled/appstore ==> | == django.wsgi ==> | == settings.py ==> | == urls.py ||
         ||             |                               |                    |                    |            ||

1. An HTTP request is made to the Apache server.

2. Apache looks in /etc/apache2/sites-enabled to see how to handle the request. The appstore configuration file is set to handle requests made to http://apps.cytoscape.org.

3. appstore tells Apache to use mod_wsgi. mod_wsgi runs a Python interpreter within Apache. appstore tells mod_wsgi to start Python with /var/www/CyAppStore/django.wsgi.

4. django.wsgi starts the Django library. It also tells Django the location of settings.py, which Django needs to start the site.

5. settings.py contains the location of urls.py (defined in the ROOT_URLCONF variable), which is a list of URLs (in the form of regular expressions) and the Python functions that handle them.

6. urls.py in the top directory of the App Store merely imports additional URLs from each Django app. It dispatches the request to the appropriate function that is designated to handle requests for a given URL. Functions are defined in the views.py file in each Django app.

7. The handler function returns with a processed HTML page.

Debugging

1. /etc/apache2/sites-enabled/appstore

   • This file tells Apache and mod_wsgi where to find the site. The most important line is this:

     WSGIScriptAlias / /var/www/CyAppStore/django.wsgi

     This tells Apache and mod_wsgi where to locate the site code. Make sure the path to django.wsgi is correct.

2. /var/www/CyAppStore/django.wsgi

   • This file invokes Django's WSGI handler. It needs to correctly reference settings.py to start the site. Make sure these two lines are correct:

     SITE_PARENT_DIR = '/var/www'
     SITE_DIR = filejoin(SITE_PARENT_DIR, 'CyAppStore')

     To check if these variables are being defined correctly, you can launch a separate Python interpreter and enter these lines:

     from os.path import join as filejoin
     SITE_PARENT_DIR = '/var/www'
     SITE_DIR = filejoin(SITE_PARENT_DIR, 'CyAppStore')

     Then check if the variables SITE_PARENT_DIR and SITE_DIR are correct.

3. /var/www/CyAppStore/settings.py

   • This file is pretty complicated. But if you've checked everything at this point, here's some ways to pinpoint problems in settings.py.

     1. If you're getting an HTTP 500 error, you can get the stack trace by turning on debug mode then reloading the page. Note that debug mode exposes sensitive information about the site to the public. Make sure to keep debug mode off as much as possible. Change to following line to True:

        DEBUG = False

     2. You can poke at the code by running a Python shell. Enter this command at the shell prompt in the same directory as settings.py:

        python manage.py shell

        You can check to see if the site's code is working correctly without having debug mode on. For example, to see if the list of all apps is working, enter this into the Python interpreter:

        from apps.models import App
        App.objects.all()

     3. The SQL database settings are specified by the DATABASES variable:

        DATABASES = {
                'default': deploy_database
        }

        Make sure that 'default' is pointing to te correct dictionary:

        deploy_database = {
                'ENGINE':   'django.db.backends.mysql',
                'NAME':     ...
                'USER':     ...,
                'PASSWORD': ...
        }

     4. If you're getting database errors, enter this command at the shell prompt in the same directory as settings.py:

        python manage.py dbshell

        If you're able to get a SQL prompt, that means Django can connect to the SQL database.

     5. If you make changes to a Python file but you're not seeing the changes taking effect, you may have to delete all the .pyc files. To do so, type this:

        make svnclean

Tips

• You can reindex the text search engine with this command:

  make index

• If you want to remove unused tags, authors, and media files, type this command:

  python manage.py garbage_dump

  If any tags or authors were removed, you will have to reindex the text search engine.

• When a developer adds a 2.x plugin, it goes into http://cytoscape.org/plugins/plugins.xml but not automatically into the App Store. To add a newly added plugin in plugins.xml to the App Store, do the following:

  1. Download plugins.xml and open it.

  2. Find the name of the plugin you want to add. It is enclosed in the <name> tag.

  3. Run this command:

     python manage.py add_plugins_from_plugins_xml name-of-plugin

     The name of the plugin must match what's in the <name> tag. If the name has spaces, enclose the name in quotation marks. Do not run this command on plugins already with app pages, as it will overwrite its details.

  4. Reindex the search engine so the app can found through free text search.

App Store Test Protocol

1. Does the front page load?
2. Does "All Apps" load?
   1. Does sorting work as expected?
3. Does a category load?

4. Does a 2.x plugin page load? (e.g. /apps/psicquicuniversalclient)

   1. Does plugin downloading work?
   2. Does "Search for posts" work?
   3. Does ratings work?

5. Does a 3.x app page load? (e.g. /apps/cluego)

   1. Do release downloads work?
   2. When a release is downloaded, is it reflected in the stats page?
6. Does app page editing work?
   1. Icon?
   2. Any link field?
   3. Details editing?
   4. Deleting a 3.0 release?
      1. Does the release still show in the download stats?
      2. Does the release no longer show in the app page?
      3. Does the release no longer show in the backend?
      4. Does depending on the deleted release fail?
      5. Does submitting a jar with the same name and version succeed?
7. Does search work?

8. Do app author pages work? (e.g. /apps/with_author/John%20"Scooter"%20Morris)

9. Does the "About" page load?
10. Does the "Contact Us" page load and work?

11. Does the /backend/all_apps page work?

    1. Does it refer accurately to icon and release URLs?
12. Do the admin pages load?

App Submission Test Protocol

App submissions will principally test the manifest file. The following snippets are for jars created with the manifest files listed below.

To create an empty jar with just a given manifest, first make sure to have an empty file in the current directory. This is because jar requires at least one input file (besides the manifest) to create a jar. Run this command to create the jar:

jar cmf manifest jar-name.jar empty

Empty manifest test

This should fail: no Cytoscape-App-Name in the manifest.

Simple app test: no version

This should fail: no Cytoscape-App-Version in the manifest.

Manifest:

Cytoscape-App-Name: blah

Simple app test: no API compatibility

This should fail: no Cytoscape-API-Compatibility in the manifest.

Manifest:

Cytoscape-App-Name: blah
Cytoscape-App-Version: 1.0

Simple app test: new app

• This should succeed and require adding a new app page.
• After submission, the admin should receive an email about the submission. It should be listed in the Pending Apps page.
• Go back to the "Confirmation" page, then click "No, cancel it". It should no longer be listed in Pending Apps.
• Resubmit the app, and accept it. It should successfully create an app page.

• An email from the account in conf/emails.py/EMAIL_ADDR should have been sent to the app author confirming the app.

• The new app page must have the submitter's email address listed as an editor.

Manifest:

Cytoscape-App-Name: blah
Cytoscape-App-Version: 1.0
Cytoscape-API-Compatibility: 3.0

Simple app test: new release

This should succeed and not require admin approval. It should still send an email to the admin about the submission.

Manifest:

Cytoscape-App-Name: blah
Cytoscape-App-Version: 2.0
Cytoscape-API-Compatibility: 3.0

Simple app test: no authorization

Submit the jar with the same manifest as above but under a different, non-admin account. Submission should be rejected.

OSGi bundle: no name

This should fail: no Bundle-Name.

Manifest:

Bundle-SymbolicName: blah

OSGi bundle: no version

This should fail: no Bundle-Version.

Manifest:

Bundle-SymbolicName: blah
Bundle-Name: blah

OSGi bundle: no Cytoscape imports

This should fail: no Cytoscape packages in Import-Package.

Manifest:

Bundle-SymbolicName: blah
Bundle-Name: blah
Bundle-Version: 3.0
Import-Package: xyz,abc

OSGi bundle: no Cytoscape version in imports

This should fail: no Cytoscape packages in Import-Package.

Manifest:

Bundle-SymbolicName: blah
Bundle-Name: blah
Bundle-Version: 3.0
Import-Package: org.cytoscape.a, org.cytoscape.b

OSGi bundle: new release

This should succeed.

• - The minimum Cytoscape version must be 3.5.

Manifest:

Bundle-SymbolicName: blah
Bundle-Name: blah
Bundle-Version: 3.0
Import-Package: org.cytoscape.a;version="(3.0,4]",
  org.cytoscape.b;version="(3.5,4]"

OSGi bundle: export package

This should succeed and ask for a pom and Javadocs.

Manifest:

Bundle-SymbolicName: blah
Bundle-Name: blah
Bundle-Version: 4.0
Import-Package: org.cytoscape.a;version="(3.0,4]",
  org.cytoscape.b;version="(3.5,4]"
Export-Package: blah

OSGi bundle: deleted dependency

Delete an existing release. Then try to submit an app that depends on it. This should fail.

Manifest:

Bundle-SymbolicName: blah
Bundle-Name: blah
Bundle-Version: 3.0
Import-Package: org.cytoscape.a;version="(3.0,4]",
  org.cytoscape.b;version="(3.5,4]"
Cytoscape-App-Dependencies: GeneMANIA;3.0.0.beta1

OSGi bundle: multiple dependencies

Submission should succeed. After submission, GeneMANIA and ClueGO must list blah as a dependency.

Manifest:

Bundle-SymbolicName: blah
Bundle-Name: blah
Bundle-Version: 5.0
Import-Package: org.cytoscape.a;version="(3.0,4]",
  org.cytoscape.b;version="(3.5,4]"
Cytoscape-App-Dependencies: GeneMANIA;3.0.0.beta2, ClueGO;2.0.0