Add two options to the `test-backend` script:
1. verbose
If given the `test-backend` script will give detailed output.
2. no-shallow
Default value is False. If given the `test-backend` script will
fail if it finds a template which is shallow tested.
This lets us cut out the line which hard-codes how deeply nested in
the tree the `run-mypy` script is, making it simpler to borrow these
scripts in other projects.
This stub file allows us to annotate view functions using the actual
types present in the bodies of the functions, rather than everything
having the type REQ.
Previously lister.py used to check whether the exclude path is a
substring of a path being considered. So it would fail when the
exclude path is an absolute path or uses '..' or '.'.
We need to update provision.py to compile the messages files, since
they are needed for the new i18n tests. And of course we need to
include the .mo files in release tarballs; there's a bit of complexity
there around how the tarball archives are created.
This results in a substantial performance improvement for all of
Zulip's backend templates.
Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.
Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`
Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
instead of `RequestContext` object. There are two reasons to do this:
1. `RequestContext` is not compatible with Jinja2
2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
includes a custom backend and a template renderer, compressors for js
and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.
Fixes#620.
This makes mypy about 15% faster running on the Zulip codebase (from
7s=>6s on my laptop), which seems worth it for losing a couple files.
This option requires a new dependency, which we add to the
mypy-specific requirements.txt file.
Also update the mypy command line to not use deprecated argument names.
This introduces a few errors, so we exclude the relevant files to keep
the mypy output clean.
Camo is a caching image proxy, used in Zulip to avoid mixed-content
warnings by proxying HTTP image content over HTTPS. We've been using
it in zulip.com production for years; this change makes it available
in standalone Zulip deployments.
This should save several minutes off the Travis CI `production`
suite's runtime, since previously we were doing the full apt upgrade
process twice, resulting in things like multiple expensive rebuilds of
the initramfs.
Add 'six' to setup-py3k, because it is being used in tools/lister.py.
Add 'typing' to setup-py3k, so that tools/lister.py can be type
annotated in the future.
Now tools/check-py3 will by default run all fixers together. This is
quicker but doesn't indicate which fixers caused the failure. The
newly added option --find-fixers falls back to the old way of checking
each fixer separately if the quick check fails.
Fixes#710.
Move recenter_pointer_on_display, suppress_scroll_pointer_update,
fast_forward_pointer, furthest_read, and server_furthest_read to
a new pointer module in pointer.js.
The `with sh.sudo` pattern that we were using in python-sh was
deprecated, and emperically hangs on Ubuntu xenial. Since in general
the use of python-sh/python-pbs caused trouble (requiring extra
dependencies, confusing syntax), this just removes it.
We replace it with a new zulip_tools.py library function that echoes
the command line and streams the output.
We do the same to install-phantomjs so we can remove that dependency.
This reverts commit d936bf61f9.
We no longer need this since we've migrated to specifying the
dependencies in the typing module that we're actually using.
tools/travis/py3k used to always exit with exit code 0.
It should exit with 1 when fixers detect a compatibility issue.
py3k used [ -z "$failed" ] to check if there was a failure.
This is wrong, since if no failure has occured, failed=0,
and -z checks if a string is of zero length. This commit also
fixes this bug.
In py3k, "git reset --hard" was called only if
libmodernize.fixes.fix_dict_six changed files and some of those
changes are not considered false positives by py3k.
But if all of those changes are not considered false positives
by py3k, then "git reset --hard" is not called and the repository
is no longer clean.
This commit fixes this bug.
tools/travis/py3k used to only check files whose names ended with .py.
Now it also checks python scripts which don't have an extension.
It uses tools/lister.py to get a list of all python files.
This has the side effect of making lint-all check all shell scripts,
not just those under scripts/, tools/, and bin/.
[commit message expanded by tabbott]
Make module tools/lister.py which lists all files in a directory
tracked by git. This is done because lister.py will be used by other
scripts in the future which have to introspect files in the repository,
like linters, static code checkers, etc.
This should fix a problem we've been having with errors downloading
the PhantomJS packages from their original hosting service.
Eventually we should move it to an S3 bucket.
This change drops the memory used for Python processes run by Zulip in
development from about 1GB to 300MB on my laptop.
On the front of safety, http://pika.readthedocs.org/en/latest/faq.html
explains "Pika does not have any notion of threading in the code. If
you want to use Pika with threading, make sure you have a Pika
connection per thread, created in that thread. It is not safe to share
one Pika connection across threads.". Since this code only connects
to rabbitmq inside the individual threads, I believe this should be
safe.
Progress towards #32.
This automatically loads settings, zerver.models.* and
zerver.lib.actions.* when you start `manage.py shell`, which should
save a bit of time basically every time someone uses it.
Fixes#275.
Add call to tools/generate-fixtures in tools/test-backend before
starting the tests. Previously, test-backend could fail if called
after tools/test-js-with-casper had failed.
Fixes#501.
manage_args is set to a list of arguments a few lines later in the
function, making this initialization as the empty string useless and
confusing.
Discovered using mypy.
It's not clear whether this will end up being net negative in value in
the long term since it's kinda hard to understand the output, but in
the short term it should prevent regressions.
It's possible we should just eliminate this mechanism, but this fixes
a proximal problem where the multi-line get_subscribers endpoint
description was being handled wrong.
Travis CI's model of installing every version of postgres on the test
VM and then shutting all the versions other than the one requested
down seems to not work very well with doing apt upgrades. It seems
the best way to resolve this is to just uninstall the versions we
don't need.
The point of the lock is to prevent two deployments happening at the
same time and racing with each other, not to prevent doing any future
deployments after an error happens (which is what the current
implementation does in practice).
Addresses part of #208.
We ran into a bug with the Travis CI infrastructure where it postgres
9.1 is installed on the system, and so when we'd do an apt upgrade
with a new version of 9.1, the 9.1 daemon would end up getting started
and conflict with the 9.3 daemon we were trying to run.
Django's `manage.py runserver` prints a relatively low-information log
line for every request of the form:
[14/Dec/2015 00:43:06]"GET /static/js/message_list.js HTTP/1.0" 200 21969
This is pretty spammy, especially given that we already have our own
middleware printing a more detailed version of the same log lines:
2015-12-14 00:43:06,935 INFO 127.0.0.1 GET 200 0ms /static/js/message_list.js (unauth via ?)
Since runserver doesn't have support controlling whether these log
lines are printed, we wrap it with a small bit of code that silences
the log lines for 200/304 requests (aka the uninteresting ones).
Several of these rules only apply to one of Python and Javascript, and
this simplifies the logic and should make our linter code more readable.
In the process, we add support for per-rule/file pair exclusions to
handle the tab exception for codehilite.py.
The #! line processing interpreted the argument to pass to `env` as
"python2.7 -u", which obviously isn't a real program.
We fix this by setting the PYTHONUNBUFFERED environment variable
inside the program, which has the same effect.
Thanks to Dan Fedele for the bug report and suggested solution!
This test caught a few bugs where refactoring had made management
commands fail (and would have caught a few more recent ones).
Ideally we'd replace this with a more advanced test that actually
tests that the management command do something useful, but it's a
start.
The node packages 'jQuery' and 'jquery' are different--'jQuery' is the
legacy support package that is needed for Zulip so the require statements
in the tests were updated.
Travis uses node 4.0 by default and we are using 0.10, so the command to
install the correct version had to be added to the .travis.yml file.
This tests whether a new patch introduces any regressions related to
any of the Python 3 compatibility fixers we've run in the past, so
that we can make continuous forward progress on our path towards
Python 3 compatibility.
This produces error output that looks like this:
"""
Testing for additions of Python 2 patterns we've removed as part of moving towards Python 3 compatibility.
Running Python 3 compatibility test lib2to3.fixes.fix_apply
Running Python 3 compatibility test lib2to3.fixes.fix_except
diff --git a/zerver/views/__init__.py b/zerver/views/__init__.py
index b5c0102..2defd46 100644
--- a/zerver/views/__init__.py
+++ b/zerver/views/__init__.py
@@ -296,7 +296,7 @@ def accounts_register(request):
do_activate_user(user_profile)
do_change_password(user_profile, password)
do_change_full_name(user_profile, full_name)
- except UserProfile.DoesNotExist, e:
+ except UserProfile.DoesNotExist as e:
user_profile = do_create_user(email, password, realm, full_name, short_name,
prereg_user=prereg_user,
newsletter_data={"IP": request.META['REMOTE_ADDR']})
Python 3 compatibility error(s) detected! See diff above for what you need to change.
"""
Running check-templates test fails when there are 'blocktrans' tags in
django templates. The fix is to add 'blocktrans' to
is_django_block_tag function in check-templates.
With this change, we are now testing the production static asset
pipeline and installation process in a new testing job (and also run
the frontend/backend tests separately).
This means that changes that break the Zulip static asset pipeline or
production installation process are more likely to fail tests. The
testing is imperfect in that it does not have proper isolation -- we
build a complete Zulip development environment and then install a
Zulip production environment on top of it, so e.g. any apt
dependencies installed for Zulip development will still be available
for the Zulip production environment. But, it's better than nothing!
A good v2 of this would be to have the production setup process just
install the minimum stuff needed to run `build-release-tarball` and
then uninstall it / clean it up so that we can do a more clear
production installation, but that's more work.
This fixes an annoying issue where one tries to rebuild the database,
and it fails due to there being existing connections.
The one thing that is potentially scary about this implementation is
that it means it's now a lot easier to accidentally drop your
production database by running the wrong script; might be worth adding
a "--force" flag controlling this behavior or something.
Thanks to Nemanja Stanarevic and Neeraj Wahi for prototypes of this
implementation! They did most of the work and testing for this.
This fixes some issues that we've had where commands will fail is
confusing ways after the database is rebuilt because data from before
the database was dropped is still in the memcached cache.
Instead, build them automatically when provision the development
environment and in update-prod-static.
(imported from commit aac8dfeaafbe872c113e5f2b6bd8f655a1af36f2)
It doesn't have any sensitive data since that lives in a separate
configuration file, and it's potentially useful.
(imported from commit 094e315439f8bd23ad07a8c2bc7d9776c8c7f096)
The tarball build process runs in DEVELOPMENT mode, assuming it is run
on a dev VM (since then there is no /etc/zulip directory). Commit
d067bcfe9d71 made settings.py import local_settings_template.py in
DEVELOPMENT mode (then "not DEPLOYED"), not local_settings.py.
(imported from commit 9a08138d748dfca9c4ab8b366bee5c2fb96c25af)
Just importing zerver.lib.cache creates a file memcached_prefix that
is mode 0444, so we need to use -f or rm will prompt about whether to
remove it. Not sure why this is apparently a new issue.
(imported from commit 93c5140b66992339859e2b204c200d1dd7a35f2d)
This commit loses some indexes, unique constraints etc. that were
manually added by the old migrations. I plan to add them to a new
migration in a subsequent commit.
(imported from commit 4bcbf06080a7ad94788ac368385eac34b54623ce)
We don't need to check whether the user exists before creating it:
CREATE USER failing is fine.
(imported from commit e8b2bc5495e328ee30d15445a566c0edff2f069d)
If we run provision.py a second time, there will already be
zulip/zulip_test users, so the CREATE USER will fail and the password
won't get updated to the newly generated value. By creating the user
and setting the password in two commands, we allow the creation to
fail without affecting whether the password is set.
Also the quoting for updating .pgpass was wrong.
(imported from commit 5e249813c17cb4829e4e4958e92aaa30563c5f96)
Sometimes I get the error "Selected message id not in MessageList"
when running the casper tests. I think it's probably when the test
user's home view does not contain any messages.
Ideally we would fix this in a way that guarantees that we generate
whatever messages the test suite needs...
(imported from commit 51a02da612dda88d60681b9e09cd6e6a2c39a470)
Source LOCAL_DATABASE_PASSWORD and INITIAL_PASSWORD_SALT from the secrets file.
Fix the creation of pgpass file.
Tim's note: This will definitely break the original purpose of the
tool but it should be pretty easy to add that back as an option.
(imported from commit 8ab31ea2b7cbc80a4ad2e843a2529313fad8f5cf)
The old language was confusing because "the interface" could refer to something
like eth0, but in actuality refers to the IP/hostname to listen on.
(imported from commit 4f77d72a4dfcdbe7e7747c6228975aa68dfbe6ac)
It's been very buggy for a while, has limited usefulness compared with
unread counts, and profiling over the weekend indicates that it's very
slow.
(imported from commit 716fe47f2bbec1bd8a6e4d265ded5c64efe2ad5c)
This doesn't change the alerting UI logic, it just turns
alert_words_ui into a module and calls the setup code from settings.js
when the settings page is rendered.
(imported from commit 05f95383b046086641280f82f648be58688efe61)
Before this change, the way we'd strip tags of punctuation
was just sort of messed up, because we'd strip the start tags
one way and strip the end tags another, and we had conditionals
for the different flavors of tags, instead of doing the stripping
when we already knew what flavor of tag we were dealing with.
(imported from commit 60c5ebd45e21b88bbfc98ff4b43dbbc6b32b38a1)
This makes it simpler to test between two VMs by allowing you to bind to
non localhost interfaces.
(imported from commit f70755533b52ff8c49fd916941d2210fb8c33b47)
Importing zerver.worker.queue_processors (which is needed to get the list of
workers to start) is slow because it, in turn, imports a bunch of stuff. So we
move the process of starting up queue processing workers into another script
that gets started in parallel with everything else.
(imported from commit 839bada6dc7b93825c69b0d8fd9fbe2de75eabee)
Add a helper to patch_global to change a global and then reset it to the
original value after a test file is complete.
(imported from commit 1b65ff6ea8693ad61b7f18f35dafa942429252a8)
In the early days of the node tests we didn't have an index.js
driver, so each test set it up its own environment. That ship
has long since sailed, so now we just require assert in index.js
as a global and make it so that the linter doesn't complain.
(imported from commit 1ded3d330ff40603cf4dd7c5578f6a47088d7cc8)
I apparently screwed up when backing up the process_loaded_for_unread
move in a way that just lost the function.
(imported from commit 91dfcf1abc85d439274cb8b0be380e9230942ebb)
The production database has a public schema. I thought we had dropped it, but
apparently not. We should match what exists in prod either way.
(imported from commit 1bf956360029ebbd59afc3cc30fca9a859343adf)
We do this by creating a new zulip{_test}_base database that only has the zulip
schema and the tsearch_extras extension. We then use that as a template when
creating zulip{_test}.
(imported from commit 8adb4b98410e4042a0187902e89c99561eac8c8f)
These classes are in test_hooks.py now. They still run as part of
the regular suite, so this is just to make it easier to navigate the
files.
JiraHookTests
BeanstalkHookTests
GithubV1HookTests
GithubV2HookTests
PivotalV3HookTests
PivotalV5HookTests
NewRelicHookTests
StashHookTests
FreshdeskHookTests
ZenDeskHookTests
(imported from commit 26a9572dd5170f9516e739d587a119bd1f87959a)
We changed our endpoint from "get_public_streams" in August, but the
API call whitelist was not updated.
(imported from commit 293c1da8e43c24ad8188ed2096a47992ad3a2c89)
Have run-dev.py watch for template changes by calling
`./tools/compile-handlebars-templates forever`. This doesn't
have much effect until the subsequent commit, but it does
alert users to broken templates.
(imported from commit 3fa5f403cabe0057f6f43180f1d09db669d98682)
The "forever" option causes the tool to continue looking for
template changes and, when they happen, to recompile them.
(imported from commit 2fa719a205f02c7c90cc071f99252148a888654f)
Before this change, we were compiling handlebars templates but then
still sometimes used the copy of compiled.js from the previous deploy,
which made zero sense. Since compiling is super fast, we continue to
compile during every deployment, and now we actually use the results
of compiling. Forcing a recompile every time avoids pitfalls like
failing to notice deleted files or failing to notice a Handlebars
upgrade.
(imported from commit 675932428ec420bfe0fd5a5c748a85600206764c)
postgres-init-db was expecting to be located two directories below the
project root. It is only one directory below the root of the project so
remove a set of .. .
(imported from commit 228b47ac2539caf2b6cd12a1b5f399534cf0c866)
This replaces the --noworkers option; the new --minimal option
starts a couple "essential" workers.
(imported from commit 4ca08709052c47257bc0448e51760edb4969d92e)
This allows us to avoid a circular import when importing models.py
from inside bugdown for the realm-filters-from-database branch.
(imported from commit 7de85b54243132ade6818b080abdc8c5e8ad84f5)
There are now 2 cases for narrowing:
1. We narrowed, but only backwards in time (ie no unread were
read). In this case, try to go back to exactly where we were before
narrowing. This behavior is unchanged.
2. We read some unread messages in a narrow. Instead of going back to
where we were before the narrow, go to our first unread message (or
the bottom of the feed, if there are no unread messages). This is new.
This means that after catching up through the sidebar, on returning
home you'll be at the bottom of your feed.
Searching for the first unread message in a message list with 40,000
messages only takes 17ms according to:
function timeit() {
var t0 = new Date().getTime();
_.find(current_msg_list.all(), unread.message_unread);
var t1 = new Date().getTime();
console.log('Find first unread: ' + (t1 - t0) + ' ms');
}
(imported from commit 87c467578a2cced0aa976d8ae2924371b85d2445)
This basically prints out a template JSON data structure to
be used with a handlebar template that you specify on the command
line. (You can actually supply multiple files, too.)
Example usage:
$ ./tools/get-handlebar-vars static/templates/tab_bar.handlebars
=== static/templates/tab_bar.handlebars
{
"tabs": [
{
"hash": "",
"title": "",
"active": "",
"icon": true,
"data": "",
"cls": ""
}
]
}
(imported from commit d7239fcae7d94038fa0e4b34c8b1208a1070ecbb)
This uses git ls-files -m, which will show modified and added files,
but it doesn't seem to show staged files, so buyer beware.
(imported from commit 6ecc1d5ee628deae17197addf5586f1f6bcd4b9c)
If you use persistent ssh connections, ssh'ing as admin will cause a sshd
process to hang around on the server, preventing us from deleting the admin
account. Therefore, we disable persistent connections for that ssh connection.
(imported from commit 2d043768417d20ef2f12695475a20b74bf3374de)
This requires a puppet apply on each of staging and prod0 to update
the nginx configuration to support the new URL when it is deployed.
(imported from commit a35a71a563fd1daca0d3ea4ec6874c5719a8564f)
When you upload a 2nd avatar to Zulip, the URL doesn't actually
change, so even new messages can show the old avatar, if your
browser is caching. We work against the cache by having the
"stamp" argument, which we vary at reload time and also when
we upload the new avatar. The browser still benefits from
cached images as new messages come in.
(imported from commit 84869c8d7f251c9f2498026a5e9e3b2451784879)
The check-handlebars-templates script now looks at most of our
back end templates to try and find imbalanced tags. This commit
fixes a bunch of the existing templates.
(imported from commit fad4a5d85d68160370dd588b41d6f125f64d198f)
Because git < 1.8.1 ignores lines with trailing slashes and
1.8.1.1 - 1.8.1.6 ignore lines without!
(imported from commit 8139a742f4a52ccb1bce4e06fb24c9626fdb01f2)
update-prod-static needs DEBUG=False. This also replaces our
local_settings.py before generating anything included in the tarball.
(imported from commit 890cd9d1a44acfd2c20e1662e0c68132c633d1b3)
We still need it in integrations, because those don't require Python
2.7, but we don't need it in any of our code that runs on internal
servers.
(imported from commit 3c340567f1a372dcb4206c6af9a6e5e18005b1b8)
The corporate "app" is not a full-fledged Django app, but it has
a urls.py and a templates directory. This commit creates the app
and moves the jobs pages into it. Localserver deployments will
not see any of the corporate code.
(imported from commit 35889c3cf92329258c30741fdfa564769a4fac1a)
The main use-case here is for ensuring that we don't deploy to master
while doing demos.
(imported from commit d3c8ba502052b352291548200032f39e0743b774)
Run the following commands as root before deploying this branch:
# /root/zulip/tools/migrate-server-config
# rm /etc/zulip/machinetype /etc/zulip/server /etc/zulip/local /etc/humbug-machinetype /etc/humbug-server /etc/humbug-local
(imported from commit aa7dcc50d2f4792ce33834f14761e76512fca252)
Tim Abbott
Umair Khan
Conrad Dean
Greg Price
Reid Barton
Eklavya Sharma
Vishnu Ks
Vladislav Manchev
Ashish
Aristeidis Fkiaras
Varshit
Josh Mandel
Allie Jones
Kara McNair
Ahmed Shibani
Steven Oud
Anders Kaseorg
Leo Franchi
Tim Abbott
David Roe
Reid Barton
Cat Miller
Jonathan Dahan
Luke Faraone
Yoyo Zhou
Zev Benjamin
Jason Michalski
Waseem Daher
Tim Abbott
Steve Howell
Zev Benjamin
Jessica McKellar
Leo Franchi
acrefoot
Luke Faraone
Kevin Mehall