This results in a substantial performance improvement for all of
Zulip's backend templates.
Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.
Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`
Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
instead of `RequestContext` object. There are two reasons to do this:
1. `RequestContext` is not compatible with Jinja2
2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
includes a custom backend and a template renderer, compressors for js
and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.
Fixes#620.
In theory these should be the same, but in misconfigured environments
(such at Travis CI) where /etc/hosts has multiple entries for
"localhost", 127.0.0.1 is safer than "localhost".
Camo is a caching image proxy, used in Zulip to avoid mixed-content
warnings by proxying HTTP image content over HTTPS. We've been using
it in zulip.com production for years; this change makes it available
in standalone Zulip deployments.
In theory, tools like populate_db should probably be in zerver, not
zilencer, but until we migrate them out, we need to include these in
EXTRA_INSTALLED_APPS in development.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.
As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.
When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.
We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.
Fixes#441.
Move recenter_pointer_on_display, suppress_scroll_pointer_update,
fast_forward_pointer, furthest_read, and server_furthest_read to
a new pointer module in pointer.js.
Previously these were hardcoded in zproject/settings.py to be accessed
on localhost.
[Modified by Tim Abbott to adjust comments and fix configure-rabbitmq]
This is in some ways a regression, but because we don't have
python-postmonkey packaged right now, this is required to make the
Zulip production installation process work on Trusty.
(imported from commit 539d253eb7fedc20bf02cc1f0674e9345beebf48)
This makes fetching settings from here for the rabbitmq config not
work; and the main place we check for this is in initialize-database
anyway.
(imported from commit fcc6eff3882f880fdc8b127daffc9a3173f3ff20)
This also removes the convenient way to run statsd in the Dev VM,
because we don't anticipate anyone doing that. It's just 2 lines of
config to configure it anyway:
STATSD_HOST = 'localhost'
STATSD_PREFIX = 'user'
(imported from commit 5b09422ee0e956bc7f336dd1e575634380b8bfa2)
This commit loses some indexes, unique constraints etc. that were
manually added by the old migrations. I plan to add them to a new
migration in a subsequent commit.
(imported from commit 4bcbf06080a7ad94788ac368385eac34b54623ce)
We can add it back later but for now we can just stick with localhost
since that's what most people will want.
(imported from commit c5fe524282219dc62a0670f569c0cb6af04be339)
Source LOCAL_DATABASE_PASSWORD and INITIAL_PASSWORD_SALT from the secrets file.
Fix the creation of pgpass file.
Tim's note: This will definitely break the original purpose of the
tool but it should be pretty easy to add that back as an option.
(imported from commit 8ab31ea2b7cbc80a4ad2e843a2529313fad8f5cf)
It's been very buggy for a while, has limited usefulness compared with
unread counts, and profiling over the weekend indicates that it's very
slow.
(imported from commit 716fe47f2bbec1bd8a6e4d265ded5c64efe2ad5c)
Now we have 2 different Zulip apps out there, and they are signed with
two certs: Zulip and Dropbox. The Dropbox-signed apps are going to need
to be sent APNS notifications from the appropriate APNS connection
(imported from commit 6db50c5811847db4f08e5c997c7bbb4b46cfc462)
The feedback bot needs to be included in the OG_ZULIPER_EMAILS so users
can send feedback.
(imported from commit b0c3295379d0f062a912f8b6c957419335be1c48)
This needs to be deployed on both prod and lb0 to be functional
DEPLOY INSTRUCTIONS: restart carefully
(imported from commit d97a450754608357418c80e5b3c7b3bbcd1d09fb)
I don't think this matters a ton, but there's no reason for
the two muting files to come before setup.js.
(imported from commit c18231b633fba6acccaf5bc180c7605f7591462e)
This experiment has been disabled for everyone for a while: if we
bring something like this back, it is not likely to be exactly the same,
and will be different enough to require a different implementation.
As it is, the summarization code was making a few code paths (rendering
especially) more complex, and is worth removing for simplicity's sake.
(imported from commit 6ac8cdc9f7077a5a1da01ab4268aba3db0bc43f8)
Before we deploy this commit, we must migrate the data from the staging redis
server to the new, dedicated redis server. The steps for doing so are the
following:
* Remove the zulip::redis puppet class from staging's zulip.conf
* ssh once from staging to redis-staging.zulip.net so that the host key is known
* Create a tunnel from redis0.zulip.net to staging.zulip.net
* zulip@redis0:~$ ssh -N -L 127.0.0.1:6380:127.0.0.1:6379 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 staging.zulip.net
* Set the redis instance on redis0.zulip.net to replicate the one on staging.zulip.net
* redis 127.0.0.1:6379> slaveof 127.0.0.1 6380
* Stop the app on staging
* Stop redis-server on staging
* Promote the redis server on redis0.zulip.net to a master
* redis 127.0.0.1:6379> slaveof no one
* Do a puppet apply at this commit on staging (this will bring up the tunnel to redis0)
* Deploy this commit to staging (start the app on staging)
* Kill the tunnel from redis0.zulip.net to staging.zulip.net
* Uninstall redis-server on staging
The steps for migrating prod will be the same modulo s/staging/prod0/.
(imported from commit 546d258883ac299d65e896710edd0974b6bd60f8)
The file test_runner.py has our subclass of DjangoTestSuiteRunner
and various methods that help it work.
(imported from commit 8eca39a7ed3f8312c986224a810d4951559e7a8b)
Before deploying to staging, create the tutorial bot:
email: welcome-bot@zulip.com
name: Zulip Welcome Bot
(imported from commit 2f337a00ffac888b121975bdb95a89cf2f8ab3a7)
Add javascript to handle the button clicks and update the status based
on the subscribe and unsubscribe events from the server.
(imported from commit 6b9c0b40d9084e3d8b64bed701ebc786bef6d432)
The one call we were making to it was actually unnecessary since highlighting is
no longer done client-side.
(imported from commit 7644a5961ca48cd57f4b65c6f698083677e0a130)
Make our dev setup more similar to prod by using compiled.js,
instead of AJAX-ing templates on the fly and compiling them
with non-node code. This will make our dev environment more
consistent with prod (to avoid surprising bugs), plus it should
be faster (fewer AJAX calls).
This change also means we don't have to keep two copies
of static/third/handlebars/handlebars.js around.
(imported from commit d8d584b9aa13adcdcce7e424033610d77d2df79b)
Otherwise the app would crash when it tried to build template context:
Traceback (most recent call last):
[...]
File ".../zerver/context_processors.py", line 23, in add_settings
'email_gateway_example': settings.EMAIL_GATEWAY_EXAMPLE,
File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 55, in __getattr__
return getattr(self._wrapped, name)
AttributeError: 'Settings' object has no attribute 'EMAIL_GATEWAY_EXAMPLE'
(imported from commit 6cc5d3f150326a55230ea91e6c228ae9b1e19df3)
We now have 2 variablse:
EXTERNAL_API_PATH: e.g. staging.zulip.com/api
EXTERNAL_API_URI: e.g. https://staging.zulip.com/api
The former is primarily needed for certain integrations.
(imported from commit 3878b99a4d835c5fcc2a2c6001bc7eeeaf4c9363)
Many deployments will not set AUTH_LDAP_BIND_DN , because they allow
anonymous binding. It is better for us to use AUTH_LDAP_SERVER_URI as a
signal for whether to use LDAP, since any deployment that uses LDAP will
set that local setting.
(imported from commit ad70bedfb572b42a9df954819593e2678729647b)
If authoritative data is available from say the LDAP database, we now
ignore the POSTed user name, and don't offer it as a form field.
We fall back to giving the user a text field if they aren't in LDAP.
If users do not have any form fields to fill out, we simply bring them
to the app without the registration page, logging them in using a dummy
backend.
(imported from commit 6bee87430ba46ff753ea3408251e8a80c45c713f)
The latter doesn't depend on the former; we can still fill in your full
name even if you didn't authenticate via LDAP.
This commit requires django_auth_ldap to be installed. On Debian
systems, you can do so via APT:
sudo apt-get install python-django-auth-ldap
On OS X, use your favourite package manager. For pip, I believe this
will work:
pip install django_auth_ldap
django_auth_ldap depends on the "ldap" Python package, which should be
installed automatically on your system.
(imported from commit 43967754285990b06b5a920abe95b8bce44e2053)
This could potentially help with debugging exactly what happened with
some issue down the line.
(imported from commit cc7321d742875b644d4727a084b462dcd01dcf10)
This is useful for the occasional case where we cannot figure out what
is causing a particular problem, but it can be easily reproduced on
staging.
(imported from commit 8b51184a8b686814f2c6ff103ba355538463ceb0)
CUSTOMER13 doesn't want it, and there's currently no nginx config
or configurable Camo URI, so it wouldn't work if image preview
were enabled.
(imported from commit 615d4a32acbc4d4d590f88cf4e7d45d8f49db1d3)
It throws an exception when the mixpanel library is not present. This
exception breaks emoji autocomplete, among other things.
(imported from commit 6ae2a69d40282701b8717b60d887836416c85a6c)
This command should be run continuously via supervisor. It periodically
checks for new email messages to send, and then sends them. This is for
sending email that you've queued via the Email table, instead of mandrill
(as is the case for our localserver/development deploys).
(imported from commit a2295e97b70a54ba99d145d79333ec76b050b291)
ScheduledJobs with type Email displace the usual mandrill codepaths
in the Zulip Enterprise deploys
* Email-specific helper functions will appear in deliver_email.py
* 0058_auto__add_scheduledjob.py
(imported from commit 8db08d8a279600322acfdbed792dc1a676f7a0ab)
update-prod-static needs DEBUG=False. This also replaces our
local_settings.py before generating anything included in the tarball.
(imported from commit 890cd9d1a44acfd2c20e1662e0c68132c633d1b3)
This was recommended by:
https://docs.djangoproject.com/en/dev/howto/deployment/checklist/
Since we don't change our deployments without restarting Django and
don't use any custom template magic, this should be a free performance
win.
(imported from commit fd498ab97d0669c3a14b342b2d2f01994a1f1ee1)
I figure we can start with 600s as a maximum age -- our threads do
many dozens of requests per minute, so I figure we'll get most of the
benefit of permanently persisting connections this way. I could also
be convinced to do just 60s, though the impact will likely to be less
visible on staging. 600s seems to be what Django originally had for
this parameter before they disabled it by default. See:
https://groups.google.com/forum/#!msg/django-developers/rH0QQP7tI6w/yBusiFTNBR4J
for discussion, which also suggests we might have issues with
runserver that we should watch out for.
(imported from commit 0ae09fa4f1b39cc88c76fa58258aaf20ab168dcf)
Move commands related to stats collection and reporting from
zilencer to analytics. To do this, we had to make "analytics"
officially an app.
(imported from commit 63ef6c68d1b1ebb5043ee4aca999aa209e7f494d)
Run the following commands as root before deploying this branch:
# /root/zulip/tools/migrate-server-config
# rm /etc/zulip/machinetype /etc/zulip/server /etc/zulip/local /etc/humbug-machinetype /etc/humbug-server /etc/humbug-local
(imported from commit aa7dcc50d2f4792ce33834f14761e76512fca252)
This moves the list of removed files from .gitattributes to
tools/build-local-server-tarball because static/ and tools/ are
necessary for update-prod-static, and it seemed best to keep the
entire list in one place.
(imported from commit 2a447cbde29e90d776da43bb333650a40d4d363c)
This is something we'll want to do before we switch to Django 1.6,
since it's the default there, and I'm not aware of any significant
problems this switch will cause.
(imported from commit fadea41f09e2179735328f99a3ec55b46c92041e)
Looking at the historical data, fewer than 50% of active users have
completed the checklist, which means that it is just persistent
clutter. We also have other better ways of encouraging people to send
traffic and get the apps now.
This commit removes both the frontend UI and backend work but leaves
the db row for now for the historical data.
(imported from commit e8f5780be37bbc75f794fb118e4dd41d8811f2bf)
The moved files are:
humbug-server
humbug-local
humbug-machinetype
Their new names are their old names with 'humbug-' removed.
zulip-puppet-apply must be run before this commit is deployed
(imported from commit f4eb523244d3409b5809c279301225d3fdf0c230)
Subclasses of QueueProcessingWorker that don't override start() will
have their consume() functions wrapped by consume_wrapper(), which
will catch exceptions and log data from troublesome events to a log
file.
We need to do a puppet apply to create /var/log/zulip/queue_error.
(imported from commit 3bd7751da5fdef449eeec3f7dd29977df11e2b9c)
This commit must be simultaneously deployed on both staging and
prod0. It also requires completely taking down the app.
To deploy these changes, do:
* check out this commit at /root/zulip on postgres0, postgres1, staging, and prod0
* stop the process_fts_updates job on postgres0 and postgres1
* stop the app on staging and prod0
* do a puppet apply on postgres0, postgres1, staging, and prod0
* move the new client certificates into place on staging and app
* move the new server certificates into place on postgres0 and postgres1
* reload the database config on postgres0 and postgres1 (this might
actually require a restart)
* run tools/migrate-db on postgres0 as root
* do a deploy through this commit on staging and prod0
* start the process_fts_updates job on postgres0 and postgres1
* do a puppet apply on nagios
(imported from commit 819bdd14326c1425e2d3041a491a8ca3b9716506)
Here we introduce a new Django app, zilencer. The intent is to not have
this app enabled on LOCALSERVER instances, and for it to grow to include
all the functionality we want to have in our central server that isn't
relevant for local deployments.
Currently we have to modify functions in zerver/* to match; in the
future, it would be cool to have the relevant shared code broken out
into a separate library.
This commit inclues both the migration to create the models as well as a
data migration that (for non-LOCALSERVER) creates a single default
Deployment for zulip.com.
To apply this migration to your system, run:
./manage.py migrate zilencer
(imported from commit 86d5497ac120e03fa7f298a9cc08b192d5939b43)
Before deploying this commit, the following commands must be run:
# rabbitmqctl add_user zulip $(tools/get-django-setting RABBITMQ_PASSWORD)
# rabbitmqctl set_user_tags zulip administrator
# rabbitmqctl set_permissions -p / zulip '.*' '.*' '.*'
(imported from commit 76d66fa6ac69aa39c91f55b9b5d5a954f9e94d73)
New dependency: sockjs-tornado
One known limitation is that we don't clean up sessions for
non-websockets transports. This is a bug in Tornado so I'm going to
look at upgrading us to the latest version:
https://github.com/mrjoes/sockjs-tornado/issues/47
(imported from commit 31cdb7596dd5ee094ab006c31757db17dca8899b)
This may require just doing an mv on the home directory, plus changing
the home directory in /etc/passwd. It should of course be done carefully.
(imported from commit 660997d897ee6d33563af74f0fc5d4267a911755)
This has been deployed, so the current code is now properly
minified, and we don't need the force_minify flag any more.
(The flag is needed when you remove a JS file from JS_SPECS.)
(imported from commit 8b3c7ffdbc875011d59c2560034750f0077db616)
The minify logic doesn't have an easy way to detect that you
removed a file since the last deployment.
(imported from commit 50d05fcdad382a586073c06d29d279433d1bba81)
We are still showing the same data points, but the logic to drill
down on details for a particular realm is now all server side,
not client side, and we are smarter about omitting fields. In
summary mode, we don't show empty Name or Email columns. In
detailed mode, we show the realm as a headline instead of a column.
In this version you do lose the ability to see all system users in
the same view, but Waseem is ok with this.
(imported from commit edd2e646ab4cf5783ea64232d0cd621debece8d4)
When you load the activity report, it will just show summary
counts for realms, but if you click on a realm, you will see
details about users in the realms. You can also click "Show all"
to see an interleaved view of realms and users.
(imported from commit b106557b1fae64d525071afc124b5a8aed319086)
I believe this was the source of the "Handlebars.compile is not a
function" errors, though I don't know why so few users got it.
(imported from commit f4a72e35bf25f679461d3c9e6938d118fec30278)
This shows up when you're not running a Zephyr mirroring bot and lets
you use Webathena to have us run it. Obviously needs more docs.
Current problems include:
* supervisorctl reload ends up recreating /var/run/supervisor.sock
with the wrong permissions, so it only works once in a row before
you need to chmod that.
* /etc/supervisor/conf.d needs to be humbug-writeable; this is a clear
local root vulnerability
* This uses SSH and thus is kinda slow.
(imported from commit 7029979615ffd50b10f126ce2cf9a85a5eefd7a2)
This change will allow us to test the muting feature on
staging. Any topic named "muted" will automatically be
muted. You can also mute any other topic on the console:
muting.mute_topic('devel', 'ios');
current_msg_list.rerender();
More UI around this experiment will be coming soon, as well
as support for muting entire streams.
The muting module keeps track of which topics are muted, but a
user can expand muted messages, and once that happens, the
messages are marked with the "force_expand" flag that gets
persisted to the back end.
Muted messages are rendered in similar fashion to the summarized
rows, and as part of unifying some of that code, we have
made it so that expanding a summarized section doesn't remove
individual flags related to summaries; instead, the messages
get the force_expand flag set.
(imported from commit acee4190e63813d46850415c41ff8ebfae4a6953)
We now show a list of users and allow you to deactivate a user using the
same process as `python manage.py deactivate_user`.
We add a new menu item accessible from the gear icon which will eventually
have much more than just this, but we have a good start here.
Here we also add a property to UserProfile which determines whether you're
eligible to access the administration panel, and then have code which shows
the menu option if so.
This introduces a new JS file, admin.js.
(imported from commit 52296fdedb46b4f32d541df43022ffccfb277297)
We instead implemented the ~desired functionality here using the
API and a bot to make a totally read-only, static, slowly-updating
view into the Zuliverse.
This is the moral equivalent of reverting deb035b4c702fcdb0e660ed549fe74c682abb6d9
(imported from commit 9d743fe82f197b37f005e5a038f77cc4b8566024)
ALLOW_REGISTER was no longer being used in determining whether you could
register for the app, so I've removed it to avoid additional local-dev /
production issues.
This closes#1613.
(imported from commit c928c6d350602d35f745ae1e60d734e4567885fc)
1) The class Filter now lives in its own module.
2) The function canonicalized_operators() is now a class method on Filter.
3) The function message_in_home moved to filter.js and became private.
4) Various calling code had to change, of course.
5) Splitting out Filter helped simplify a few tests.
(imported from commit e41d792b46d3d6a30d3bd03db0419f129d0a2a7b)
The compose_fade has three public exports:
set_focused_recipient
unfade_messages
update_faded_messages
All code was pulled directly from compose.js, except for the
one-line setter of set_focused_recipient. The focused_recipients
variable that used to be in compose.js was moved to compose_fade.js,
hence the need for the setter.
(imported from commit 462ca5d0d0bd58612d0197f3734a8c78de8c6d30)
"Kiosk mode" is a "read-only" Zulip suitable for embedding into
an iframe on another site. I say "read-only" in quotation marks,
because the account is still a fully-fledged active account on
the server, and we just tear out a bunch of stuff in Javascript
(that a malicious user could easily re-enable).
So in that sense, it's not actually safe in security-sensitive
environments -- malicious users logged in via kiosk mode
can do anything the kiosk-mode user can do.
(We need this functionality for the customer3 realm specifically;
we'll possibly just tear this code back out once that experiment
has run its course.)
(imported from commit deb035b4c702fcdb0e660ed549fe74c682abb6d9)
This is a pure refactoring that mostly just moves code from
subs.js to the new stream_color.js and updates module references
accordingly. In order to prevent introducing some exports,
update_stream_color was given an additional "sub" parameter
and update_stream_sidebar_swatch_color was given an "id"
parameter.
Killed off unused initial_color_fetch var.
(imported from commit b7644ce67f50d31fb46f564d758d661eea776aa6)
This includes a hack to preserve humbug/backends.py as a symlink, so
that we don't need to regenerate all our old sessions.
(imported from commit b7918988b31c71ec01bbdc270db7017d4069221d)
Umair Khan
Tomasz Kolek
Vishnu Ks
Tim Abbott
goelakash
Eklavya Sharma
Alexander Trost
Javier Ros
Chris Chapman
Allie Jones
Shumbashi
Tim Abbott
Reid Barton
Yoyo Zhou
David Roe
Anders Kaseorg
acrefoot
Cat Miller
Zev Benjamin
Leo Franchi
Jason Michalski
Luke Faraone
Steve Howell
Waseem Daher
Allen Rabinovich
Leo Franchi
Zev Benjamin
Jessica McKellar
Luke Faraone
Kevin Mehall
acrefoot
Jessica McKellar
Scott Feeney
Steve Howell
Tim Abbott