This section briefly summarises the functionality and design of the new architecture. Naming
a top level “module” that is “core” to do site wide stuff, such as “soc/user/create” generic urls such as ‘program/create’ are removed and replaced by e.g. ‘gsoc/program/create’
At a pinch we could just go with the naming without introducing a communication structure. However this is a route to spaghetti code and great ambiguity in meaning of objects. For example, what does ‘App_task_limit’ mean? It could depend on the context, GHOP or GSoC. If we restrict the communication to well designed channels, i.e. if we genuinely keep the design modular, these kinds of problems are less likely.
In the future we might have the following modules (this is not a roadmap, but is meant to give an impression what kind of modules we want to support).
issue tracker module ? what would it be used for ?
For example, there are no issue trackers “out there” that allow you to group issues under a main issue, currently we mark those issues as Duplicate, while instead they should be grouped. Also, rather than having a ‘comment based’ issue tracker, a ‘wiki based’ issue tracker with voting support would be a lot nicer. As said, this is not a roadmap, so I'm not saying this should be implemented by us, but I know that others would want to implement it :).
This section describes the conversion process from the pre-module architecture to the new module architecture.
Estimated is that the implementation of a basic framework as described in this document will take a full weekend (with no other obligations) by two developers.
I think it will take longer but we can bet :-) and work syncing of those two developers will be also a problem. How are you going to divide work on this new architecture between two developers ?
As you can see from the code below most of the design is already done. Dividing the work is easy, each developer focusses on one method of the API (e.g., one working on sidebar, one on sitemap).
The full conversion of the current system to the new architecture (e.g., so that the core uses the core callback to register it's sidebar and sitemap entries, etc) is estimated to take a week (with no other obligations) by two developers.
Core: The code module that runs when Melange starts up. It is responsible for invoking any other modules. It contains a built in ‘module’ with name ‘noplugin’ that does things we currently expect to be required in every program. It may turn out that later some of these move out into modules, but that is not planned at this stage.
Module: A related body of code that can register its services and requirements with ‘Core’.
The order in which the core invites registrations from the modules could be important at a future date. For GHoP and GSoC the order is unlikely to be important. We can address the issue of ‘order’ at a later date.
The precise details of what we want to register are also likely to grow over time. This is our ‘plug in API’. This is the correct way to develop - organically. We have a structure in which this can happen.
By means of the ‘register capability’ and ‘require capability’ calls described below modules can ensure that after all modules have loaded successfully each module has it's required capabilities satisfied. This way there is no need for modules to depend on one another, and order does not matter.
This section contains some general code snippets that are related to the new architecture.
Some server side code. This lives in the core.
def hasCapability(callback, capability): """Checks whether a callback support the specified capability. Args: callback: the callback to check if it supports the capability capability: the capability to check for """ if not hasattr(callback, capability): return False func = getattr(callback, capability) if not callable(func): return False return True
Some callback API code. Variants of this live in each module.
class Callback(object): """Callback object that handles interaction between the core and a module. """ def __init__(self, core): """Initializes a new Callback object for the specified core. """ self.core = core self.core.registerService(...) def registerWithSitemap(self): """Called by the server when sitemap entries should be registered. """ # register our sitemap entries self.core.registerSitemapEntry(...) ... def registerWithSidebar(self): """Called by the server when sidebar entries should be registered. """ # require that we had the chance to register the urls we need with the sitemap self.core.requireService('registerWithSitemap') # register our sidebar entries self.core.registerSidebarEntry()
In order to get everything running build.py will be extended to load all modules defined in settings.MODULES. The MODULE_FMT is ‘soc.modules.%s’ for Melange. The module list will be stored in a global and used from a number of places in core for different capabilities. There will also be a subroutine for calling the callbacks for a particular capability (not shown).
One consequence of the design is that at some date in the future, if a capability has been superseded, the core no longer needs to support it. It will not ask for that capability and callbacks will not need to supply it.
Core itself should also be part of the callbacks and use the module API to do stuff.
def getModuleCallbacks(): """Retrieves all callbacks for the modules of this site. callbacks for modules without a version number or the wrong API_VERSION number are dropped. They won't be called. """ fmt = settings.MODULE_FMT modules = [__import__(fmt % i, fromlist=['']) for i in settings.MODULES] return [i.getCallback() for i in modules]
add in somewhere... core.triggerCapability(‘registerWithSitemap’)
Currently many models are shared while they should not be.
These are all in core
Looks like these need a base class and then to have specialised GHOP and GSoC versions.
org_app.py (Might be replaced by surveys)
Converting to the new module system is going to be a multi-step process. This way we can have more control over the data. We will also be keeping all the old data until the conversion process is complete.
The steps are outlined below:
Convert the current Program entities to GSoCProgram entities keeping the same keyname.
Convert all Organization entities to GSoCOrgs keeping their repsective keynames but with updating the Program reference (scope) to the respective GSoCProgram. The GSoCProgram can be easily looked up since it will have the same keyname as the Program entity.
Update all OrgAdmins to GSoCOrgAdmins and change their scope to reference to the new GSoCOrganization and the program reference to the new GSoCProgram.
The same as 3 but just for Mentors -> GSoCMentors.
Convert the Student entities to GSoCStudent entities and update their Program reference to contain the new GSoCProgram.
At this point we are done with creating new entities, however several references still need to be updated
Update the Review entities. We should look into deleting the role reference property or otherwise we need to ensure that we convert the Students to GSoCStudents, Mentors to GSoCMentors and OrgAdmins to GSoCOrgAdmins.
Update the StudentProject entities with the following references:
Update the ProjectSurveyRecords and GradingProjectSurveyRecords org property to reference the new GSoCOrganization entity.
Update the GradingSurveyGroup model with a reference to the new GSoCProgram.
Update the Document model:
- For prefix == ‘org’ the scope should be updated to contain the new GSoCOrganization entity. If home_for is set this should be updated to the new GSoCOrganization as well.
- For prefix ==‘program’ the scope should be updated to contain the new GSoCProgram enity. Same goes for the home_for property.
The move to the new module system is complete
Change proposal to have the organization as scope and student as a separate property. This would require updating the references from the reviews.
Might also be good to move Reviews to an id_based system as well as Comments. This will help with change 12 as well as make it a more generic system.
Rework document system to contain the prefix internally and work on an id-based system. This will make it easier to move documents between scopes and also makes it easier for the system for use within the module approach.
Before we embark on this journey of change literally the largest parts of our data store it might be worthwhile to discuss how this process is executed.
As you can see there are clean steps in this process and that is something that can be useful when changing the data. It gives one a point of rest to check wether or not the conversion is complete and it is something that can be done again if necessary.
My proposal would be to create a Conversion module with a View and appropriate conversion tasks. The view has one simple page which shows you the conversions you can execute when you update from version X. The conversions are all accompanied by a button which sends POST?? data to the view that translates it into the correct task that needs to be fired, fires that task and the conversion is under way.