#summary Guide on tips and tricks for writing tests for Melange
#labels Contents-Draft,Importance-Useful,Phase-QA

This page is inteded for tips and tricks for writing tests for Melange.

<wiki:toc max_depth="3" />


= Writing tests =
== Testing framework==
In a nutshell, tests are all about comparing the expected result with the actual result after application of a function/method under test. If you expect to get a particular value, you can assertEqual to the value of the actual result. If you expect that a particular exception should be raised, you can assertRaises the exception. If you expect that a template is used to render a web page, you can assertTemplateUsed the template. If you expect that a mail has been sent to a particular person, you can assertEmailSent(to=the_persons_email_address). On the other hand, if you cannot predict the exact value but you expect that the value will change after the application of a function/method under test, you can assertNotEqual to the value before the application of a function/method under test. In addition, you may also want to assert that the environment before your test is your expected/required testing environment.

Not only should normal situations/use cases be tested but also should other possible situations/use cases, which is particularly important for security tests. For example, we should not only test that an HTTP POST with a correct XSRF token can be successfully applied, we should also test that an HTTP POST with an incorrect XSRF token should be forbidden. For anther example, for views that are only accessible to users with developer privilege, we should not only test that the views work if the current user is a developer, we should also test that the views are forbidden if a normal user or even an unregistered user tries to access the views.
=== `Python unit test` ===
All existing logic and models tests of Melange are built on Python unittest TestCase (unittest.TestCase). It gives you several assert methods as well as setUp and tearDown methods which will be respectively called before and after each test case, and also enables your tests to be run by almost all python test runners.
==== `Usage` ====
  * Derive your test class from unittest.TestCase.
  * Use setUp method to set up test environment, e.g. seed data store.
  * Use tearDown method to clean up the test.
  * Use assertTrue, assertEqual, assertNotEqual, assertRaises and so on to assert that the expected result actually happens.
  * Check [http://docs.python.org/library/unittest.html the documentation of Python unittest] for more information.
  * Example usage in Melange: tests.app.soc.logic.`*`, tests.app.soc.modules.gsoc.logic.`*`, tests.app.soc.modules.ghop.logic.`*` 
=== `Django TestCase` ===
Most views tests are built on Django TestCase (django.test.TestCase) which extends Python unittest TestCase by adding extra functions, e.g. Django test client for view tests which works with GAE/Melange and fixtures which does not work with GAE without any modifications, patches or helpers.
==== `Usage` ====
  * Derive your test class from tests.test_utils.DjangoTestCase.
  * Use self.client.get or post to get the response of requesting a URL.
  * Use assertTemplateUsed, assertTemplateNotUsed, assertContains, assertEual and so on to test your expectation.
  * For HTTP POST tests, an XSRF token may be needed, which helps prevent cross-site request forgery attacks. Use getXsrfToken method to return an XSRF token for request context signed by Melange XSRF middleware and then add this token to POST data in order to pass the validation check of Melange XSRF middleware for HTTP POST.
  * Check [http://docs.djangoproject.com/en/dev/topics/testing/#module-django.test.client the documentation of Django test client] for more information.
  * Example usage in Melange: tests.app.soc.views.`*`, tests.app.soc.modules.gsoc.views.`*`, tests.app.soc.modules.ghop.views.`*` 
=== `MailTestCase` ===
All mails related tests are built on gaetestbed.mail.MailTestCase which collects all mails sent and can check if emails satisfying some criteria are sent.
==== `Usage` ====
  * Derive your test class from tests.test_utils.MailTestCase.
  * You can use assertEmailSent or assertEmailNotSent to assert emails satisfying certain criteria are or are not sent out.
  * *Note:* Of these criteria, to, sender and subject are exactly matched while body (text body) and html (HTML body) are partly matched.
  * You can also specify an optional argument n to assert exactly n messages satisfying the criteria are sent out.
  * Check [http://github.com/jgeewax/gaetestbed/tree/master/gaetestbed/ the source code of gaetestbed] for more information.
  * Example usage in Melange: tests.app.soc.tasks.`*`, tests.app.soc.modules.gsoc.tasks.`*`, tests.app.soc.modules.ghop.tasks.`*` 
=== `TaskQueueTestCase` ===
All task queues related tests are built on gaetestbed.mail.TaskQueueTestCase which can check if task queues satisfying some criteria are spawned.
==== `Usage` ====
  * Derive your test class from tests.test_utils.TaskQueueTestCase.
  * You can use assertTasksInQueue(n=None, url=None, name=None, queue_names=None) to assert that there are n tasks with task URL equal to url, task name equal to name using any queue in queue_names which is a list of strings.
  * Check [http://github.com/jgeewax/gaetestbed/tree/master/gaetestbed/ the source code of gaetestbed] for more information.
  * Example usage in Melange: tests.app.soc.tasks.`*`, tests.app.soc.modules.gsoc.tasks.`*`, tests.app.soc.modules.ghop.tasks.`*` 

== Naming convention ==
The name of the test module for testing a codebase module is the name of the codebase module plus a prefix of "test`_`" (without quote), e.g. test_base.py. The name of the test class for testing a codebase class is  the name of the codebase class plus a surfix of "Test" (without quote), e.g. BaseTest. The name of the test method for testing a codebase method is the name of the codebase method with the first letter capitalized plus a prefix of "test" (without quote), e.g. testGetForFields.

== Organization of tests ==
All tests are in a separate directory
{{{
MELANGE-ROOT/tests
}}}
The organization of the tests for a module within the directory is the same with the corresponding module in the codebase but with a "test`_`" (without quote) prefix before the module name. For example, the tests for the module app.soc.logic.models.base are placed in the module tests.app.soc.logic.models.test_base.

== Utilities ==
All are in tests.test_utils module.
=== `MockRequest` ===
This is a shared dummy request object to mock common aspects of a request similar to Django test client.
=== `StuboutHelper` ===
This is a helper class using the the StubOutForTesting class of pymox. It is mainly used to stub out/ replace a function or a method of the codebase with a dummy one during the test. 
=== `DjangoTestCase` ===
It extends Django TestCase in order to extend its functions as well as remove the functions which are not supported by Google App Engine, e.g. database flush and fixtures loading without the assistance of Google App Engine Helper or patches for Django. It can be used to do view tests in the replacement of Python unittest TestCase because of its extra assert methods and test client for view tests.
=== `MailTestCase` ===
It extends gaetestbed.mail.MailTestCase by subclassing unittest.TestCase so that all its subclasses need not subclass unittest.TestCase in their code and by overriding assertEmailSent method to extend its functions.
=== `TaskQueueTestCase` ===
It extends gaetestbed.taskqueue.TaskQueueTestCase by subclassing unittest.TestCase so that all its subclasses need not subclass unittest.TestCase in their code.
=== `Data seeder` ===
Data are usually needed to conduct tests. The seed functions are here to seed data for tests. 
  * There are two seed functions, seedn and seed. The difference between them is that seedn seeds n data of the model class while seed only seeds one data of the model class.
  * Location: soc.modules.seeder.logic.seeder.logic.Logic
  * Function prototype: seedn(self, model_class, n=1, properties=None), seed(self, model_class, properties=None)
  * Usage: any number of properties can be specified either with their values or with the data provider used to generate the values. Unspecified properties will be generated randomly; unspecified ReferenceProperty will be generated and seeded recursively.
    * model_class: data store model class
    * n: number of entities to seed
    * properties: a dict specifying some of the properties of the model_class objects to be seeded. The key of the dict is the name of the property. The value of the dict is either the value of the property or the data provider used to generate the value of the property, e.g.
{{{
        {"name": "John Smith", 
         "age": RandomUniformDistributionIntegerProvider(min=0, max=80)}
}}}
  * For more detailed usage and examples, please check their documentations and their tests in tests.app.soc.modules.seeder.logic.seeder.logic.SeederLogicTest

== Tools ==
=== pymox ===
  * http://code.google.com/p/pymox/ 
  * Pymox can be used to isolate objects from other interacted objects by mocking the behaviours of those interacted objects.
  * It is in MELANGE-ROOT/eggs now.
=== fixture ===
  * http://farmdev.com/projects/fixture/
  * Fixture can be used to load and reference preset test data.
  * It is in MELANGE-ROOT/eggs.
=== Django test client ===
  * http://docs.djangoproject.com/en/dev/topics/testing/#module-django.test.client
  * It acts as a dummy web browser which simulates HTTP requests on a URL and returns the response.
  * It is mainly useful for Django view tests.
  * It is in app.django.test.Client.
=== gaetestbed ===
  * http://github.com/jgeewax/gaetestbed/tree/master/gaetestbed/ 
  * It is very handy to do mails and task queues related tests with gaetestbed.
  * It is in MELANGE-ROOT/eggs.
=== gaeftest ===
It is in MELANGE-ROOT/eggs.
=== zope.testbrowser ===
It is in MELANGE-ROOT/eggs.

= Running tests =
== Running all the tests ==
Suppose that the current working directory is the root of your local clone of the Melange project. You can run all the tests by running the following command
{{{
bin/run-tests
}}}

== Running options ==
If you want to get test coverage report, add option --coverage, i.e.
{{{
bin/run-tests --coverage
}}}
The default covered package is the whole soc package (--cover-package=soc.).

Internally, it uses nose test runner. So, all the options of nosetests should also work here. For example, you can use  -e REGEX or --exclude=REGEX to exclude tests that match the regular expression REGEX. E.g. to ignore the functional tests run:
{{{
bin/run-tests -e test_functional
}}}
*Note*: prebuilt nose library (MELANGE-ROOT/eggs/nose-0.11.3-py2.5.egg) does not include the coverage plugin, so you cannot use the -with--coverage option to get test coverage report.

== Running the tests in one module ==
If you want to just run all the tests in one module (say test_base), execute
{{{
bin/run-tests tests/app/soc/logic/models/test_base.py
}}}
or
{{{
bin/run-tests tests.app.soc.logic.models.test_base
}}}

== Running a group of test cases in one module ==
To just run a group of test cases in one module (say test_base.BaseTest), execute
{{{
bin/run-tests tests/app/soc/logic/models/test_base.py:BaseTest
}}}
or
{{{
bin/run-tests tests.app.soc.logic.models.test_base:BaseTest
}}}
== Running a single test case in one module ==
To just run a single test case in one module (say test_base.BaseTest.testGetForFields), execute
{{{
bin/run-tests tests/app/soc/logic/models/test_base.py:BaseTest.testGetForFields
}}}
or
{{{
bin/run-tests tests.app.soc.logic.models.test_base:BaseTest.testGetForFields
}}}

*Note:* although running only one test module or a group of test cases can save much time, it is desirable to run all the tests at least once before merging your code just in case it breaks others' code.


= The test environment =
The test environment is set up by the tests/run.py file. It adds libraries to Python path, sets up Python environ variables, GAE datastores, memcache etc, creates a Melange core, registers callbacks, and runs tests through nose.
== User account ==
  * apiproxy_stub_map.apiproxy.RegisterStub('user', user_service_stub.UserServiceStub())
  * The email of the user account of the test environment is test@example.com and the user is logged on automatically if the user account (test@example.com) is in Melange User datastore.
== Datastore ==
  * apiproxy_stub_map.apiproxy.RegisterStub('datastore', datastore_file_stub.DatastoreFileStub('test-app-run', None, None))
  * It is a memory based datastore which is cleared automatically after each test case, so it is not necessary to clear it manually after each test.
== Memcache ==
  * apiproxy_stub_map.apiproxy.RegisterStub('memcache', memcache_stub.MemcacheServiceStub())
  * It keeps all data in the local process' memory.
== Mail service ==
  * apiproxy_stub_map.apiproxy.RegisterStub('mail', mail_stub.MailServiceStub())
  * It does not really send email,  but only logs a description of the email to the developers console.
== URL fetch service ==
  * apiproxy_stub_map.apiproxy.RegisterStub('urlfetch', urlfetch_stub.URLFetchServiceStub())
  * A Python httplib based stub version of the GAE urlfetch API .
== Task queue service ==
  * apiproxy_stub_map.apiproxy.RegisterStub('taskqueue', taskqueue_stub.TaskQueueServiceStub(root_path=yaml_location))
  * Unlike the mail service, the task queue service can actually run tasks if auto_task_running is set to True (False by default).