Forum OpenACS Development: Customizations

Collapse
Posted by Malte Sussdorff on
We have to deal with customizations frequently. A method I have been using is to make the customizations using parameters, callbacks and as a last resort "forks" from the original code.

Nima brought to my attention a different way of doing this and I wondered if this would be useful for OpenACS. He can explain this probably much better than me, but the general idea would be the following.

In addition to *.tcl or *.adp allow a *-custom.tcl and *-custom.adp. The request processor would then be smart enough to serve the "-custom" file if the "-custom" file exists instead of the normal one. So if we have index.tcl and index.adp we could make our customization by copying index.tcl to index-custom.tcl (same for .adp) and prevent the fork. An update from OpenACS would then update the original file, but we could retain our version and do a manual diff to decide if we want the latest features or not.

What do you think about this method?

Collapse
2: Re: Customizations (response to 1)
Posted by Dave Bauer on
I proposed another method which is similar but allows cutomization per package instance if necessary.

Basically you create a custom directory under your openacs install and can put any custom pages there in the same path as under packages. ie: custom/forms/www/index.adp

This is better if you use CVS since you don't overwrite your changes.

You an even do custom/www/package-instance/index.adp to customize per instance.

This would require a small change to the request processor. One interesting twist would be to use the regular tcl files under packages, but find an adp under /custom if it was there.

Collapse
3: Re: Customizations (response to 1)
Posted by Nima Mazloumi on
No matter how we do it I believe we really need it. I personally prefer the first solution because I can directly see which packages are customized since I only work in a directory.

Two additional thoughts:

1. the naming should be a parameter
2. the customization should be configurable on subsite level. Maybe we provide a default word like 'custom'.

This would allow a single installation of OpenACS to serve different customized versions. The request processor can check for each subsite whether customized pages exist and serve them.

Another extension to 2. would be to lookup customized files automatically. Thus if I have a subsite foo and a subsite bar then the request processor checks for *-foo.tcl and *-bar.tcl file to serve.

I agree to Dave that we need to have customization on instance level. Thou I believe that on subsite instance level is enough.

Collapse
4: Re: Customizations (response to 1)
Posted by Dave Bauer on
Nima. I like the per subsite customization. That works for me.

I think your idea is much simpler and will work well. ie: page-name-subsite.tcl could be the default. Any reason to not use the URL of the subsite and forget using parameters?

I can see turning OFF this feature to speed up the request processor, but I don't think it makes sense to bother making the suffix configurable.

You'd need access to the filesystem to create the custom file and there's little chance someone will create a file that matches the pattern accidentally.

Collapse
5: Re: Customizations (response to 4)
Posted by Malte Sussdorff on
How about using the subsite_id instead? This is not going to change for your custom site, even if you rename your subsite. I would just have one additional keyword like "custom" which is applicable to all pages regardless of subsite.

Having an "off" switch will most certainly help. Caching does not make any sense in my opinion. But to speed things up I would use a delimiter not commonly used in URLs, like "index+custom.tcl", so the RP can very quickly check if there is a file that matches this pattern in the file system (if we were to use "-" this will bring issues with index-2 and so on and so forth)

Collapse
9: Re: Customizations (response to 5)
Posted by Emmanuelle Raffenne on
Hi,

I don't like using subsite_id since it will be different in each installation, by installation I'm thinking about development and production servers of the same site.

Collapse
6: Re: Customizations (response to 1)
Posted by Nima Mazloumi on
subsite_id has the benefit that it is unique and will never change but it is not human readable. But why not.

I see three different ways:

- suffix-file.tcl
- file-suffix.tcl
- file.tcl-suffix

What do you think is best?

Collapse
7: Re: Customizations (response to 1)
Posted by Dave Bauer on
file-suffix.tcl

I don't like using subsite_id. How will I know which file is which when I am in emacs? Since you'll need to use a regexp anyway I don't think using the ID is any benefit.

If you have multiple subsites with the same name, they'd need to be nested so you could do

file-path-to-subsite.tcl

which is kind of verbose but much more clear what you are actually editing.

Collapse
8: Re: Customizations (response to 1)
Posted by Nima Mazloumi on
What if the path to the subsite changes? Maybe we can extend site-map to auto-rename all files, then using the path to the subsite is fine.
Collapse
10: Re: Customizations (response to 1)
Posted by Don Baccus on
To try to figure out where this discussion is starting from (i.e. base assumptions), let me make sure you all know that the request processor will try to resolve the file first in the file system, then the site map, right?

i.e. /www/foo.tcl will automatically override /packages/acs-subsite/www/foo.tcl.

This is only helpful if you know your site map but discussion should begin with this mechanism in mind. Before adding new ones ...

It would be nice to have a dynamic mechanism to augment the static one that already exists but ummm ... I'm not sure like the approach discussed above.

Collapse
11: Re: Customizations (response to 10)
Posted by Don Baccus on
Hmmm ... actually malte's original post adds nothing to the capability that already exists ... this makes me think that perhaps people aren't aware of the mechanism that already exists.

Daveb's talking about a per-instance mechanism. That also exists today, but only if you know the instance name, which is clunky. However ... the proposed mechanism is also clunky, no?

Collapse
12: Re: Customizations (response to 11)
Posted by Don Baccus on
A good example of an openacs-4 site that makes good use of the existing feature can be found at:

http://openacs.org

:)

Collapse
13: Re: Customizations (response to 11)
Posted by Malte Sussdorff on
Don, which mechanism are you talking about? If you refer to your clunky way of using /www/forums/message-view and /www/malte2/forum/message-view and /www/forum3/message-view, that is not an option for my original request, as I would have to create the custom file in the file system for every request done by the request processor.

As mentioned before, I need customizations for package key, for *ALL* instances of a package. I do agree that the whole idea of subsites which came afterwards does not make any sense in this light and should be dumped.

Collapse
14: Re: Customizations (response to 13)
Posted by Don Baccus on
OK, it wasn't clear from your first request that you were talking about multiple instances (and since daveb said "my idea also works for multiple instances" ... I interpreted "also" literally, or perhaps misread).

However, I'd rather build on the existing facility which searches an existing directory structure than add a second facility that looks for individual files buried in the package with magic suffixes.

The general solution should be obvious, one I've mentioned many times in other contexts, that packages should be first-class objects (i.e. there should be a package type, and packages subtyped from the package type). Integrating packages into the object scheme would mean that, for instance, you could derive malte-forums from forums, the RP could search malte-forums/www/* and if a file's not found there, go up the type tree until the template's found.

That, in my book, would count as a generalization of both the package scheme and the existing RP scheme.

Now, that's a dream that ain't coming true soon, but can we come up with a clunky kludge that's more inline with a "real" solution to package customization than using magic filename suffixes known to the RP?

Even something like <customizes package-key="forums"> in a private package, with that private package being mounted and the RP working with files in that private package just as it does now with URL-related paths (/www, /package-instance etc) would at least be moving in a more reasonable direction, no?

Or "<customizes-ui ...", in fact we've had discussions about making ui packages in the past ...

I don't pretend the above suggestions to be sufficient, just trying to trigger some reasonable thinking here based on the notion of package derivation rather than a convention based on magic file-name suffixes ...

Collapse
15: Re: Customizations (response to 14)
Posted by Malte Sussdorff on
Agreed, the idea of subtyped packages is the right direction. So let's focus on this one a bit more.

Is there a way to prevent us from having to make apm_package_types objects?

a) Add a supertype to the table
b) Work with package_ids, forcing the supertype package to be mounted at least once.

Both do not appeal to me but maybe someone else has some more ideas on that.

What would be involved in making apm_package_types objects? I remember there was a document somewhere on how to make your package use acs_objects, but I forgot where it is. Anyone? And is it that much work that we should shy away from it?

Here are some ideas:

- Create an object_type package_type which uses the apm_package_types extended table
- add a package_type_id which references acs_objects
- drop the PK constraint on package_key, just make it unique and not null
- create objects for all package_types of the object_type "package_type"

- We do not need to change any existing select scripts as we are adding a column, not taking one away. Or am I missing something here ?
- We need to change any script that inserts into apm_package_types, which means to rewrite the function apm_package_type__create_type. And that should be it.

I would think about the RP later, but what are the reasons that speak against making package_types acs_objects ?

Collapse
16: Re: Customizations (response to 15)
Posted by Tom Jackson on
What exactly is meant by customization? A long time ago I worked on a site, mydomain.com. The owner/manager had lots of ideas, one was running many similar sites, but with highly customized services. For instance a full featured service would offer DNS, email, web redirection. Besides the feature diffences, each service could have different backend configurations, like separate nameservers or email servers.

Of course, each site would appear as a separate website with custom look.

What a nightmare it would be to churn through all this for slight differences, or merely to avoid security issues. But what about custom pages, new pages for additional functionality?

Anyway, the requirements and my own laziness lead to the development of a single set of tables for all sites, a single instance of ACS4, and an administrative interface which allowed for the instant creation of a new service at the data model level.

Certain things setup in advance greatly help with customization, but the biggest are:

* design for customization
* independent customization code for data model
* complete separation of data from presentation code
* customization aware request processor.

(Shorter version: existence, knowledge and application of tools for Model-View-Controller pattern)

(Even shorter version: integrated separation)

Basically this all boils down to the fact that customization is not an easy add-on. Also inherent in customization is that it is nearly impossible to know in advance what is going to be customized. If customization could be handled by a simple handful of package parameters, it would fall under the heading of configuration, that is, what is allowed is completely defined in advance by the package writer. If this is offered up as customization, developers will quickly argue for new parameters to take into account their changes.

Collapse
17: Re: Customizations (response to 16)
Posted by Malte Sussdorff on
Customization here is for those cases where OpenACS does not allow me to work in a separate package or change templates or parameters or callbacks to my liking and where adding functionality to the checked out package to achieve this is not desirable. Basicly anytime you change something in acs-core and do not want to commit it back.
Collapse
18: Re: Customizations (response to 17)
Posted by Tom Jackson on
Customization here is...anytime you change something in acs-core and do not want to commit it back.

This is essentially a fork and is already fully supported. This type of customization could be characterized as changes which were not planned for and which nobody wants in the core package.

Instead of asking for changes to support your specific proposed method of customization, why not first change your own request processor and see how well it works for you?

The rp is already complicated and this feature would probably need some examples to show how it works, how it should be used and when. But the proposal appears to require replacing any file which needs any change with a new slightly different file. But for some reason you wish to retain the option of switching back to the original after an update and manual compare/diff. I wonder how the rp will figure out how to ignore the custom files, or which ones to ignore, since an adp and tcl could be changed. Best to not speculate until there is something working and the process can be tested as a whole.

Collapse
19: Re: Customizations (response to 18)
Posted by Malte Sussdorff on
"why not first change your own request processor and see how well it works for you"

Because I try to collect feedback from the community before I head in any specific direction, especially when it comes down to coding changes into core. And as you can see from the discussion it sparked, my "implementation" would not have been one generally acceptable.

In the past the change I thought off would work when you wanted to change the login screen for users in acs-subsite/lib/login.adp. I managed to make the location of this file a parameter in 5.4 and now you can store that file in a custom package. I did it this way because none of my clients wants the default login.adp, therefore general usefulness is implied.

But what if a client has changed /acs-admin/www/admin/become.tcl to make an additional permission check for special sysadmin privs which they have in their custom application. None of my other clients wants this. It is not generally useful in OpenACS (I'd assume), so the only options at the moment I have is:

a) Make the location a parameter (yet another one?)
b) Write a callback so others can make additional checks if the want to
c) Rewrite their whole handling of permission checks for sysadmins to just use ACS permissions (they would not pay for that)
d) Fork and always remember there is a change in that file when merging new versions from OpenACS
e) Write /www/acs-admin/admin/become.tcl

a-c) Do not make sense.

d) Is something I would like to avoid and be it for the reason that the client could by accident say "update from repository" and overwrite the change.

Doing d) has the benefit that, when a new version comes out and you use a clever merging tool then chances are that you will not run into any problems and still retain your custom changes. At least for a minor change like this. But what if your whole page has changed? Then it might be better to have a custom version of the file and just look at the changes in the OpenACS version and modify if needed manually.

e) Works beautifully (thanks Don for reminding me about that), but only in packages that you mount *once* (otherwise you would have to do it for every mountpoint).

I have seen client installations where a custom package was used and they included the OpenACS functionality. That works as well, but provides a lot of code and makes code maintenance more difficult (as you need to copy files which are not really needed as well, just to make the custom package provide the minimal functionality needed). At least I initially got confused which package is used and mounted when working on that site.

So the idea of Don's appeals to me most at the moment and if there is the time I will probably investigate it a little bit more. For the time being and my immediate needs I am fine with option e), though I would like to come up with a general plan (best practice) on how to do these kinds of customizations / forks you name it.

Collapse
20: Re: Customizations (response to 1)
Posted by Malte Sussdorff on
Matthew described yet another way on how to do customizations:

"One of the ways that I have done this in the past is to create a "shell" package. i.e. lets say project manager is heavily customized for the client. Then you create the package client-project-manager with www/index.vuh that defaults to converting the url reference to all files in project manager. Then for those files you need customized you can tell that custom packages index.vuh to reference the custom version instead. "

Collapse
21: Re: Customizations (response to 20)
Posted by Tom Jackson on
Very good idea. Does this require copying both tcl and adp files, plus their support files, or how does it work?
Collapse
22: Re: Customizations (response to 1)
Posted by Dave Bauer on
Tom,

The index.vuh method is very handy.

Basically you'd copy any files you wanted to customize and edit them in your custom package.

Malte,

You shouldn't need to "tell" the index.vuh to serve the custom files since the RP will find them before getting the index.vuh.

Collapse
23: Re: Customizations (response to 22)
Posted by Don Baccus on
"One of the ways that I have done this in the past is to create a "shell" package. i.e. lets say project manager is heavily customized for the client. Then you create the package client-project-manager with www/index.vuh that defaults to converting the url reference to all files in project manager. Then for those files you need customized you can tell that custom packages index.vuh to reference the custom version instead. "
This is a stripped-down variant of the notion I have of (eventually) being able to declare packages subtyped from parent packages, using an index.vuh file to serve either the custom or generic package files manually.

This gives you a way to do the customization that doesn't require hacking core with a kludge, or waiting until perhaps (someday??? never? :) something more generalized is implemented.

This shouldn't require mounting of the parent package (as I imagine implementing derived packages this wouldn't either), but rather your custom package. As dave says, the RP will find your custom templates in the mounted custom package. Your index.vuh file can then manually return the /packages/parent-package/www/* template file. If the parent package is installed but not mounted the tcl libraries will be loaded, so you'll be set, no?

If you try this out and it works well for you, tell us all about it :)

Collapse
24: Re: Customizations (response to 23)
Posted by Gustaf Neumann on
the s5 package is an example of an implementation of a derived (subclassed) package. The package s5 (http://media.wu-wien.ac.at/download/s5-0.4.apm ) is derived from xowiki. See http://www.openacs.org/forums/message-view?message_id=980065 for the original posting. i have not done any work to subclass package parameters.
Collapse
25: Re: Customizations (response to 23)
Posted by Tom Jackson on
The CAMS package demonstrates this very well.

The idea could be easily applied to any OACS package with a few small edits to existing scripts and an index.vuh file to handle the details. This is different than what I have discussed before.

Example1: http://rmadilo.com/files/cams/www/one/

The index.vuh examines the url and chooses the template to use. The following urls are the same, the first goes through index.vuh:

http://whogeenie.com/cams/one/524/

http://whogeenie.com/cams/one/one?content_type_id=524

Example2: http://rmadilo.com/files/cams/www/archive/

The index.vuh file does a little more here. The url is again examined, but the url parts are turned into a date and put into the form with rp_form_put. Then rp_internal_redirect is used to go to the real page, which could be accessed directly.

So the point here is that the passed in urls don't match any files, so they get caught by index.vuh. So in an OACS package, you could put a url part that indicated which instance or subclass was being accessed. This could be used to rp_internal_redirect to the correct tcl file...or with a few more changes to the tcl file, you could just specify the template.

What would be needed is the index.vuh file to use rp_form_put to insert a template path/name. Then the tcl page would need to end with ad_return_template $template. This editing could be done by the lone developer, or become part of the package at some point. Another possibility is a simple unix trick. Instead of editing the tcl file, just add the index.vuh file which redirects to a special name:

rp_internal_redirect customerA-one

Since the Tcl file doesn't need changing (in this example), just make a symlink from one.tcl to customerA-one.tcl and add a new template file customerA-one.adp.

This means that the template file must be stored in the same directory, which might not be okay, or maybe not. Maybe the index.vuh file could redirect to the templates directory and the symlink could go to there. Worth a try.

I think a fully supported method would be a little more tidy, but would probably require a few core changes. These ideas can be tested and integrated with current packages as needed by end users or package maintainers.