collective.twitterportlet
so we created a
quick (and slightly dirty?) fix to get them up and running again:
edition1.twitterportletfix.
The obvious symptom we were confronted with, was that there were no tweets shown in the portlet. Instead, it contained the following text:
There was an error while rendering the portlet.
And the error log included entries like this:
AttributeError: 'unicode' object has no attribute 'get'
After vaguely recalling something about a Twitter API change, we started looking around and found the Twitter blog entry where they stated that the API v1 is retired and we should use API v1.1. But API version 1.1 requires you to use OAuth.
The good news is that
collective.twitterportlet
uses python-twitter. And that
package is compatible with version 1.1 of the Twitter API. So upgrading to
version 1.0 or newer of python-twitter
, should at least make the Twitter API
wrapper compatible.
The bad news is that just upgrading python-twitter
is not enough. You
still need to give it some keys (consumer key, consumer secret,
et cetera) to get data from Twitter. Otherwise, you’ll see these kind
of messages in you logs:
TwitterError: [{u'message': u'Bad Authentication data', u'code': 215}]
In my humble opinion the optimal solution would be to do something similar as the combination of collective.facebook.portlets and collective.facebook.accounts provides. That is: allow the user to configure accounts (and perhaps even applications?) and let the user choose which account to use per portlet.
Unfortunately this involves more work and time than we could spend at the moment with customers waiting for their home page to look good again.
Due to the time constraint we decided to create a small package,
edition1.twitterportletfix,
that would solve our immediate need. This package allows the user to configure
the required keys/tokens. It also customises the Renderer
class of the portlet
to use those keys to call the Twitter API.
Obviously this package will not fulfil all needs (and I’m not really proud of it) but for our use cases it should be enough for now. At least until there is a better solution available.
As I already stated, this is not an optimal solution and I’m not really proud of it. So why did I still go ahead with it? There are several reasons.
First of all, even if I would have liked to contribute, the PyPI page of collective.twitterportlet does not list a publicly available repository. Looking around on the collective organization on GitHub also did not turn up anything. I did find a repo with the same name but saw no relation between the owner of the repo and the owner of the package.
Furthermore, even if I had found a repository, contributing to the original package would have delayed this fix. My current solution does not provide a fix for every use case of the package. For instance, you cannot have two portlets showing different accounts. But building that proper solution (and effectively replicating the functionality of collective.twitter.accounts) would have taken more time and I needed a fix for our customers sooner rather than later.
(My focus was on fixing collective.twitterportlet
rather than
replacing it. As a result I failed to look around more and did not
know about collective.twitter.accounts
until Héctor commented on this
article. Otherwise I would have tried to have
collective.twitterportlet
use collective.twitter.accounts
, instead of
reinventing the wheel.)
So I was fully aware that edition1.twitterportletfix
is not a
permanent solution. That is why I tried to make sure that you can
cleanly uninstall the package. Once there is a better solution
available—or you decide to replace collective.twitterportlet
completely—you should be able to uninstall the fix and not leave a
trace.
A couple of years ago I created mr.inquisition to get more insight in the content of a foreign Zope Object Database (ZODB). And while I believe it still may have its uses (although I haven’t personally used it for a while), you may want to start off with the following command:
$ bin/zopepy -m ZODB.scripts.analyze var/filestorage/Data.fs
This results in for example:
Processed 10611 records in 150 transactions
Average record size is 571.80 bytes
Average transaction size is 40449.41 bytes
Types used:
Class Name Count TBytes Pct AvgSize
------------------------------------------ ------ -------- ----- -------
AccessControl.users.User 2 262 0.0% 131.00
App.ApplicationManager.ApplicationManager 1 107 0.0% 107.00
App.Product.ProductFolder 1 34 0.0% 34.00
BTrees.IIBTree.IIBTree 302 64876 1.1% 214.82
BTrees.IIBTree.IITreeSet 1952 114421 1.9% 58.62
...
webdav.LockItem.LockItem 22 5817 0.1% 264.41
...PersistentAdapterRegistry 3 13840 0.2% 4613.33
zope.ramcache.ram.RAMCache 1 288 0.0% 288.00
========================================== ====== ======== ===== =======
Total Transactions 150 39.50k
Total Records 10611 5925k 100.0% 571.80
Current Objects 6286 2696k 45.5% 439.25
Old Objects 4325 3228k 54.5% 764.46
Thanks to an article by Nejc Zupan from about a week ago (Dexterity vs. Archetypes) in which he used this—at least for me—hidden gem.
]]>While I could access Whiskers via https://whiskers.example.com, references in the HTML to the favicon and the CSS were to http://whiskers.example.com/static/… And that either generates a warning about unsafe content or the browser might decide to not load the assets at all. And especially the missing CSS was severely impacting the usability.
First I tried to solve this in Whiskers itself. But I soon discovered
that the master.pt
template in Whiskers contains several calls to
static_url,
for instance:
<link rel="stylesheet" href="${request.static_url('whiskers:static/css/bootstrap.css')}" ... />
<link rel="stylesheet" href="${request.static_url('whiskers:static/whiskers.css')}" ... />
And those resolved to http://whiskers.example.com/static/… so I had to convince Whiskers (or actually Pyramid) that we were using SSL. As a result my next attempts involved changing the Apache configuration. But after trying several options I could not get it working (possibly also due to an older version of Apache).
So I left the configuration unchanged:
<VirtualHost <ip>:443>
... basic stuff about the server name, logs and SSL certificates ...
RewriteEngine on
ProxyPreserveHost on
# We use a custom CSS file.
Alias /static/whiskers.css /var/www/whiskers/static/whiskers.css
RewriteRule ^/static/whiskers.css - [L]
RewriteRule ^(.*) http://127.0.0.1:6543$1 [P]
<Location />
AuthName "Whiskers"
AuthType Basic
AuthUserFile /path/to/htpasswd
require user spam eggs ham
</Location>
<Location /buildouts/add>
Satisfy Any
</Location>
</VirtualHost>
After stumbling on a link to
the reverse proxy section of the Waitress documentation
I decided to try a different approach. My production.ini
was
basically a copy from the
example on GitHub
and it contained this section:
[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 6543
I installed Waitress in my virtualenv and replaced the above section with this:
[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543
url_scheme = https
And now the CSS is properly loaded!
I don’t know if there are easier/better ways to solve this, but this works fine for us.
]]>Diazo is for (re)theming websites. Not just Plone sites, it works on any technology. Diazo combines the HTML of the website—the content—with the HTML of the theme provider—the theme—to deliver a themed website.
With Diazo there is a natural separation between the HTML delivered by the designer or frontend developer and the logic (calculations, data, etc) which is usually handled by the backend developer. It adds an additional level of abstraction to the stack. To put it differently: Diazo offers an interface between the developers and the designers, while keeping a clear separation of responsibility. Because the developer does not have to transform the HTML from the designer into (TAL) templates, this also removes a step from the process where normally errors can creep in.
Diazo can be put in front of your existing website; the technology used for the website does not matter. The existing website is still there. You can have the multiple designs for a single website at the same time. You can even use Diazo to only apply the theme when you are on a sub site.
Diazo uses XSLT but, theoretically, you do not have to write XSLT yourself. Which is good because XSLT is hard to read. If you do write XSLT, don’t forget to place comments so a month from now you still understand what you wrote today.
When you change the Plone output so it delivers what the Diazo theme expects, you have to write less complex rules and require as little XSLT as necessary. You can also create a theme that takes the output of Plone into account so less transformation is required.
Speaking of those rules: for an overview of available rules and the
order in which they are processed (for instance <drop />
is
executed before <replace />
), see the
Diazo documentation.
The package
plone.app.theming
provides an interface to modify themes. It allows you to edit your
rules.xml
file and the rest of the them from within Plone. By using
the inspectors you can also select elements in your theme and content
(Plone site) and have the editor create a rule for you. (From what
I’ve seen during the workshop you likely want to tweak the rule
somewhat but it’s a starting point.)
It is possible to set hostnames that use the unthemed version of
the site in the advanced settings of the theming control panel. By
default 127.0.0.1
is listed there.
A Diazo theme expects a specific layout. Obviously you’ll need to have
your theme: an HTML file (index.html
for example) and accompanying
assets (CSS, JavaScript, images). Besides those files, you also need
to have the files manifest.cfg
, rules.xml
and an image with a
preview of the theme, preview.png
.
An example of a manifest.cfg
file:
[theme]
title = Demo Theme
description = My first Diazo theme
preview = preview.png
For details on the content of the manifest.cfg
file, check out the
manifest file section
plone.app.theming documentation.
A very basic rules.xml
file to get started:
<?xml version="1.0" encoding="UTF-8"?>
<rules
xmlns="http://namespaces.plone.org/diazo"
xmlns:css="http://namespaces.plone.org/diazo/css"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<theme href="index.html" />
</rules>
The <theme href="..." />
line makes sure that the index.html
file
from the theme is used. Otherwise you would still see the Plone theme.
Note that you can split up the rules into several files. For instance:
<?xml version="1.0" encoding="UTF-8"?>
<rules
xmlns="http://namespaces.plone.org/diazo"
xmlns:css="http://namespaces.plone.org/diazo/css"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xi="http://www.w3.org/2001/XInclude">
<theme href="index.html" />
<xi:include ref="rules_normal.xml" />
<rules if-content="/html/head/title[.='foo']">
<xi:include ref="rules_foo.xml" />
</rules>
</rules>
To make the Plone site editable in the themed site, you need to add the following to your rules:
<after
css:theme-children="html head"
css:content="html head script"
/>
<after
css:theme-children="html head"
css:content="html head base"
/>
Note that this only includes the JavaScript and the <base />
element
from Plone. To actually show the edit bar, you’ll need to add some
additional rules and also include some CSS in your theme to style it
properly.
You could also try to use plone.app.toolbar. But note that currently (plone.app.toolbar 1.1 and plone.app.theming 1.1b1) there is a small user interface problem: the toolbar is on top of the controls when you expanded the HTML mockup or unthemed content inspector.
If you do not have the Plone Toolbar installed, you can toggle the source view and full screen mode of the inspector. But when the Plone Toolbar is installed, it overlaps with these controls.
As long as the page is long enough that you can scroll down, you can do just that to reveal the controls again. (Since the controls scroll with the page while the position of the toolbar is fixed, they show up when you scroll on the page.)
Diazo is a wrapper around XSLT. It provides a seamless integration to (re)theme your website. It provides a higher level of abstraction, but you can also use XSLT as a fallback.
Although you can use Diazo without Plone, there currently is only tooling within Plone.
This was my first introduction to Diazo. My initial reaction is that I like the separation of concerns and responsibility between front- and backend. I can also imagine that it is easier to work with other parties that build the theme (HTML and CSS) and do not require in-depth Plone knowledge. I cannot yet judge on potential gains in time when comparing the development of a Plone theme with building a Diazo theme.
I am glad that you can edit the rules from within Plone and have the inspectors to help you selecting the elements. At the same time I can imagine that someone with more knowledge about Diazo, XSL and the actual output of Plone (which isn’t my expertise) rather just works on file system with their editor.
There are also a few things that we at Edition1 need to investigate and/or solve before we can even think about using Diazo for our customers. For instance, we use collective.editskinswitcher combined with collective.lineage. Our customers have a separate edit skin (with a preview of the content as the visitors of the site will see it) and can select a theme per child site. We’ll have to see how we can replicate something similar with Diazo, perhaps by using lineage.themeselection. That is: assuming we want to keep this setup—the Plone Toolbar also looks really nice!
Either way, I learnt a lot. So thanks Douwe, Kim Chee and Goldmund, Wyldebeast & Wunderliebe for organising the workshop!
]]>Paul had a project where PDFs were workflowed to several sites using MultiSite. This meant there was lots of duplication. After migrating to Plone 4 and blob storage, there was still a lot of data.
He created a script (bscompress) for:
In his case, the blob storage went from 14625 Megabytes to 2395 Megabytes.
Things still to do:
ZODB/src/ZODB/blob.py
.Dylan Jay talked about attracting more new people to get started in Plone. Dexterity and PloneIDE are great, but we need more. David Glick started to build a theme editor (plone.app.themeeditor). It uses jQuery UI and is a lot better than the Zope Management Interface (ZMI). There is going to be a PLIP about it.
Johannes made a new release of plone.app.event. Please take note of the installation instructions. They include configuring the right time zone.
Johannes did a quick demo of the package. There are still issues, but it’s safe to call it a beta version.
Johannes thanks all the contributors for their work.
Mark: note that the package was moved from the Collective to the Plone organisation on GitHub on October 13th.
All the team leaders (security team, UI team, etc) went on stage so people would know who is who. And whom to talk to if you want to get involved.
His company (4teamwork) created ftw.tabbedview. It can be integrated with collective.js.extjs, collective.quickupload (if installed) and ftw.table.
The package ftw.tabbedview
contains filtering, sorting, grouping,
managing columns, batching, flexible sources (catalog, SQLAlchemy,
…), data transformation (icon, authors, dates, …) and is highly
configurable. It can also be used on pretty much any context.
Philip helped to organise the Plone Konferenz München 2012. About 150 people attended and there were lots of new people. The goal of the conference was to attract decision makers. It was a very successful conference.
Tip: include a sprint lounge where people can sprint the whole conference. This way you can attract people that want to code and work together, but do not speak the language the conference is held in.
Such local conferences increase the visibility of the Plone community and can attract new members to join the community.
They were able to donate € 3000 to the Plone Foundation.
Solving problems in plone package at a time:
[Mark: The video of this lightning talk is definitely worth watching.]
If we have to believe the Mayan symbols, this year will be the end of the world. What better way to experience that day than by joining a sprint? So join the “End of the world” sprint in Ushuaia, Provincia de Tierra del Fuego, Argentina on December 21st, 2012.
Whiskers is a Pyramid application that collects all packages used in your buildouts. That is, if you use buildout.sendpickedversions. This way you can quickly see which buildouts use a certain version of a package. You might even be able to use this data to find out which eggs in your buildout cache are not used anymore.
A new solution to change how we collaborate: desk surfing. There is a Plone core developer dedicated desk in São Paulo. (You can also use it if you are not a core developer by the way.) If you are in São Paulo go there and contact Érico.
(Calvin Hendryx-Parker adds that you are also welcome in Fortville.)
Since Calvin missed is own presentation earlier that day, he did the three minute version of his talk “Playing With Blocks: 6 Pro Tips for Building Your Next Super-Charged Plone Site”:
The talk he would have held this morning will be recorded and put online.
STX Next created a small addon for Plone: stxnext.grayscale. This package transforms content of a page to grey scale. It has been developed for Polish user requirements related to national mourning after a tragedy has occurred. Governmental sites and bank sites change their colour scheme to grey scale out of respect. The images are cached on file system and are thus not generated each request.
A talk about amplify.cc, a prototype to replace PostNuke based community portals. It uses only one content type. They use Dexterity behaviours on content instances instead of on a class. Their conclusion is that having one content type for everything is possible and even works really nice.
A lot of countries have their own localised Plone site, e.g. Italy, the Netherlands, Brazil, Spain, France and Poland. They all look different. But by using a bar at the top of the page which links to other Plone sites in the world, we can unite those sites and make it easier to discover them.
]]>A sprint is a great way to discuss a problem with a package. If the maintainer is not at the sprint him-/herself, then usually someone involved in the package is available.
The most important rules:
Basically a sprint is a lot of people who can do anything they want. That could be a personal project, a PLIP, a package, etc. Everything is fine. You can also join a sprint topic of someone else. Just do what you want to do or find interesting. Either way: ask questions.
People always seem very busy. But people prefer helping you fixing stuff than fixing things themselves.
There’s a theory: whenever there are more than ten Plone people in a
room, they start moving the tables. :)
.
Sprinting doesn’t stop with coding. It’s the other way around: the non-coders might even be more productive. A sprint is also a great opportunity to discuss problems or approaches.
]]>This is the second half of Cris’s previous talk. This part is about using and extending ZopeSkel. By creating your own templates you can start with a customer project really fast.
Do you have code you write over and over again? Is the code almost the same each time and are the places where you change stuff always the same? You can avoid writing all that code by creating a template.
You can write the template the way you want. This is just a system. This means you can version control your own process by documenting it in a template.
The first question you need to ask yourself is “what do I want?” The
next thing you will want to investigate is what you have already got.
You can use “bin/templer --list
” and “bin/templer <template_name> --list-variables
” to find out more about the templates ZopeSkel has
to offer. Then have a look at the code provided by the template.
To wire everything up, you need to do four basic things:
Cris has already written the documentation for this tutorial. It currently lives on his laptop, but it should be posted online soon. I (Mark) decided to just write down some notes instead of trying to replicate Cris' documentation.
The related documentation can be found here:
Tips:
python setup.py develop
” (so “develop
” instead of “install
").templer
namespace; for your own project/infrastructure specific
templates, use your company’s namespace.We used to have ArchGenXML to generate code for us. But it only created (Archetype) content types. This may have been the reason for content type centric development. Later we needed eggs and other scaffolding and thus another tool.
On May 25th 2006, the first commit to ZopeSkel was made by Daniel
Nouri. From there it grew organically. Then local commands happened so
developers could enhance already created packages (for instance to
include portlets, content types). But most templates asked too many
questions. And the descriptions around the question were not always
clear. Many templates looked similar so which one should your choose?
There was way too much (repeated!) code. That code was not updated
consistently so for example one template generated a tests
directory
while another created a tests.py
file.
To have local commands, your package has to depend on Paste. As a
result, the Paste
, PasteDeploy
and PasteScript
eggs end up
inside your product. Even worse: if you are not careful, you can
accidentally add these packages to your version control system. This
is a big problem.
On the No-Fun ZopeSkel BBQ Sprint in 2009 the question “Who is ZopeSkel for?” was answered. It was decided that is it primarily a tool to help newcomers. For experienced developers it’s easier to start with empty structure and fill the bits and pieces in themselves.
The outcome of the BBQ sprint was the “bin/zopeskel
” script which:
paste create -t
” commandHowever, the existing documentation was not cleaned up. As a result
people still use the “paste create -t
” command and don’t even know
what the “zopeskel
” command is for.
Another outcome was that Cris became the official maintainer. And the plan was formed to break up ZopeSkel (see the GitHub repository which still contains a document with the reasoning behind this).
Work to break up started in early 2010. Templer provides templates in packages of related functionality: a buildout package, Zope related things, Plone related things, etc. But the commands for the user should remain the same because there’s so many documentation already available.
Main difference between the old ZopeSkel and Templer: you cannot create a shared structure in ZopeSkel. Templer provides structures, which are similar to templates but they cannot be run individually and only exist to be included in templates. For instance you can pull the structure for the requested licence when creating a package.
The new ZopeSkel version is a Templer application which bundles
stuff so you can use it as you were used to. You can just use
easy_install
or pip
to install ZopeSkel (your version of pip
needs to be new enough though).
Or put it in your development buildout by including a part which includes PasteScript and ZopeSkel, for example:
parts =
...
zopeskel
[zopeskel]
recipe = zc.recipe.egg
eggs =
PasteScript
ZopeSkel
The next talk teaches how to create templates, in the sprint following the conference you can actually create templates.
Note that Templer currently still depends on Paster and uses Cheeta templates. Crushinator wants to solve these problems. The only dependency would be a package for the templating language you want to use for your templates.
]]>David split up the changes into categories. But first something about the process.
A PLIP (Plone improvement proposal, basically a ticket) first needs to be approved by the Framework Team. Once it has been approved, implementation starts. When the code is ready the Framework Team does a review. If the Framework Team says “yes it’s okay” there is still more work to be done before it is completely integrated in Plone (like final documentation, small changes).
PLIP criteria:
plone.app.collection
yet.).zip
file with the theme which contains your rules.xml
file. You can
also edit a theme with the ACE editor. And a rule builder to spit
out the Diazo rules file.plone.app.dexterity[grok,relations]
in your
eggs. Both the grok
and relations
parts were not ready for
default inclusion but are available and useful. If you used
Dexterity already, definitely include relations
.placeholder
attribute).z3c.form
support to plone.app.portlets (Plone 4.3).<body>
. (Plone 4.3) to make theming
easier, e.g. when using Diazo.on
” instead of
using “live
” or binding by event name.Tip: upgrade to DateTime version 3.0: it uses far less memory!
plone_deprecated
skin is no longer enabled (Plone
4.3). It is still there but it is just not used by default.There are bunch of packages under consideration for Plone 4.4, for instance:
Interesting PLIPs David would like to see:
portal_actions
portal_skins
portal_css
portal_javascripts
z3c.form
instead)portal_properties
(move them to plone.app.registry
)David asks us to get involved in building the next version of Plone:
What will be the next version, 4.4 or 5? If there is backwards incompatible stuff we want to include, it’s probably going to be version 5. Then again, releasing version 5 might also ‘just’ be good for marketing reasons. In short: David also does not know yet.
]]>Ideally you run the tests of a project often (on every commit) in a clean environment. This way you can make sure that not only the product works properly, but also that the buildout of the product works on a clean system. That last part is really helpful during sprints: if someone cannot get the buildout running, it’s most likely the problem of the machine of the developer, not the product itself.
If you haven’t seen the tutorial.todoapp then definitely check it out. It shows some great best practises, including tests and CI.
Travis is free for any public GitHub repository. You can also have paid subscriptions for private builds.
Travis already ran 757k tests for +10k open source projects. They are crowd funded: +120k USD from +700 people. They support a lot of programming languages and preinstalled DBs (you only have to enable them).
Travis is very easy to setup. To get started you:
.travis.yml
file..travis.yml
file.Example of a very simple .travis.yml
file:
language: python
python:
- 2.7
install:
- python bootstrap.py
- bin/buildout
script:
- bin/test
(Note that because the tests are run from a clean Ubuntu install, you have to have a buildout configured for you project.)
With Jenkins it is harder to get started. With Travis the minimal configuration is quite simple. But you can get more complex if you want to. The drawback is that you only get the console output. With Jenkins you can get more information (e.g. coverage reports).
There are a number of services on the default Ubuntu image that is used. You just have to enable them. For instance, the X Virtual Frame Buffer.
You can whitelist or blacklist branches of your repository. This
prevents errors when you are just trying things out. If you want to
skip a specific build from being tested, you can add “[ci skip]
” to
the commit message.
Build notifications can be sent. Mail and IRC are the most common ones, but many more options are available.
There is
Travis browser extensions
that will show you the Travis status of a project when you go to the
page. If you want to, you can also create a status image
(https://travis-ci.org/[YOUR_GITHUB_USERNAME]/[YOUR_PROJECT_NAME].png
),
e.g for on your PyPI page.
You can also have pull request testing. If someone issues a pull request from a branch, Travis merges the code and you get a notification whether the pull request is good to merge because the tests succeed (or not). You can even see the status per commit (if there is some back-and-forth on an issue for example).
Limitations of Travis: there’s a 15 minute build runtime limit.1 The limit is for the entire build—from start to the end of the test. And since the build starts from scratch every time, this may be an issue. And again, the reporting is still limited.
Tips to speed up the build:
socket-timeout = 3
” and
“allow-hosts = ...
” to the [buildout]
section)Nejc would like to see Travis and Jenkins go hand in hand in Plone.
Advantages of Travis CI:
View the slides or watch the video.
Update (2021-08-02): This information also looks outdated. Please refer to the Travis CI documentation ↩︎