Apparently `manage.py collectstatic` by default strips files starting
with "." from what it collects. This is a reasonably precaution,
though mostly irrelevant to us, since Zulip primarily runs that as
part of build-release tarball, which runs in a clean directory.
It also breaks our current approach for transferring node_modules to
prod machines via release tarballs; this change fixes that bug.
- Add websocket client to create connection with SockJS websocket server.
It contains callback method to launch after connection setup.
- Add '--websocket' parameter to 'check_send_receive_time' script to
check websocket connection.
- Add testing websocket connection to production installation checking.
- Add cronjob to launch websocket connection nagios test.
This makes it possible for Zulip Nagios monitoring to check for
problems impacting the websockets sending code path, which is what all
web users use.
Previously, we got the following as a part of the output when running
`tools/run-dev.py` without provisioning:
It looks like you checked out a branch that has added
dependencies beyond what you last provisioned. Your tests
are likely to fail until you add dependencies by provisioning.
which is a bit confusing.
This finishes clearing unnecessarily excluded items from the global
tools/lint-all exclude list. The remaining items are third-party code
or generated code that doesn't make sense to lint.
Fixes#822.
This required instead exempting these files:
bots/jabber_mirror_backend.py
tools/deprecated/iframe-bot/show-last-messages
Turning on mypy for the API exposes issues in the annotations
of other files.
We used to create endpoints with Client._register.
Now we now have explicit methods for the endpoints.
This allows us to add docstrings and stricter mypy annotations.
This fix also introduces a call_endpoint() method that avoids
the need for manually building urls with API_VERSTRING when you
know the URL pattern of the endpoint you want to hit (and when
the API doesn't have a convenient wrapper).
I fixed a bug with create_users where it now uses PUT instead
of POST.
I also removed client.export(), which was just broken.
I had to change recent-messages and zulip-export, which were
using client.do_api_query and Client._register.
Now it's easier to just call client.call_endpoint() for
situations where our API doesn't have convenient wrappers,
so that's what I did with those scripts.
- Add rewriting 'X-REAL-IP' header for each request through dev
proxy server.
- Change webpack configuration ip to 0.0.0.0 to fix socket.io build
for launching remote dev server.
The following emojis are colored white (with a transparent background)
and are hard to read in the context of Zulip. This commit changes their
colors to blue. The emojis are:
hash, zero, one, two, three, four, five, six, seven, eight and nine.
This fixes#1969.
- Add spider for documentation crawler, which processes user help
documentation on test development server.
- Add tool script for help documentation checking launching.
Fixes#2639
Previously, we were doing this request to the production server before
waiting for all the supervisord processes to start; it's possible this
could cause failures where we hit the server before the Django uwsgi
processes are up.
Hopefully fixes#2723.
Most of the changes to support this were merged some time ago; what
remains are these changes:
* Update requirements.txt
* Django 1.10: Upgrade success-http-headers.txt file.
- We no longer get the absolute urls, see
https://docs.djangoproject.com/en/1.10/releases/1.9/#http-redirects-no-longer-forced-to-absolute-uris
- The headers are capitalized, previously, they were in upper case.
* Bump PROVISON_VERSION to 3.0 since this is a disruptive change.
Fixes#3.
This is a major change to the /#subscriptions page, converting it to
by a side-by-side list of streams and their settings in an overlay.
There are no new features added/removed, but it's a huge changeset,
because it replaces the old navigation logic and moves the stream
creation modal to appear in the right side of this overlay.
- Replace download-zxcvbn with downloading it from npm.
- Change zxcvbn.js path to node_modules (because npm put it to
`node_modules` directory.
- Bump `PROVISION_VERSION` in `version.py` to 2.4.
Fixes#2423.
When we migrated run-dev.py from Twisted to Tornado a few weeks ago,
the --interface argument wasn't properly ported and thus was ignored.
This restores the original functionality of defaulting to only
listening on localhost.
Ideally, we'd replace the vagrant/zulipdev user check with something
that just checks whether a special file that is created by the
Vagrant/remote-dev-vm creation process exists; that would be more
robust.
(Why is -u needed at all? I’m not sure, but test-run-dev spins forever
“Polling run-dev...” without it.)
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Previously, we didn't install/pin our own node.js version, and thus
had this wrapper script to manage it. Now that install our own node
via nvm, there's no reason we still need this.
Fixes#2409.
We now instrument URL coverage whenever you run the back end tests,
and if you run the full suite and fail to test all endpoints, we
exit with a non-zero exit code and report failures to you.
If you are running just a subset of the test suite, you'll still
be able to see var/url_coverage.txt, which has some useful info.
With some tweaks to the output from tabbott.
Fixes#1441.
This commit adds a basic eslintrc that emulates jslint defaults.
Rules that conflict with our existing code have been switched to
warnings instead of errors. Globals have been added to the eslintrc. The
bundled js file (generated by webpack) and blueslip.js are ignored with
.eslintignore.
To display warnings, run npm run lint-loud. This runs eslint without the
--quiet option on static/js and frontend_tests.
npm run --silent lint is run by tools/lint-all (in addition to jslint).
The --silent option is used to suppress the default output from npm run.
Fixes#535.
This helps automatically detect a common issue where users try to use
the Vagrant development environment on windows without administrator
permissions, which results in errors trying to create symlinks.
We set this up initially with all of the rules that Zulip violates
disabled.
Also, the pep8 linter is substantially slower than the other Zulip
linters, so we've put it behind an option to `tools/lint-all`.
These are not particularly interesting to measure test coverage for,
since migrations are not run as the part of the test suite by
construction, and management commands aren't being tested by this test
suite.
- Use tornado as proxy server for development environment,
replacing twisted (this doesn't support websockets).
- Upgrade tornado version to 4.4.1 (needs to be coupled to the
above since neither change works without the other)
This creates the new topic_list.js module, and the first
function that we extract is topic_list.update_count_in_dom().
This function needed to be decoupled from some non-topic-list
stuff which was overly complicated.
Sometimes remote servers are unavailable or on maintance
what is the cause of broken back-end tests. Such response errors
is excluded from raised exception cases and just notify about
wrong link in test log.
- Add script to compile documentation build and start crawler
to check documentation.
- Add documentation test script to backend travis test case.
- Add log level argument to test-documentation script.
Fixes#1492
- Add scrapy project with rewrited 'crawl' command and spider to check documentation.
Command was rewrited due to return exit status by the exception existing,
it returns exit code 0 if exception happens inside spider in standard behavour.
- Add scrapy requirements to for dev environment. It was added to twisted requirements list.
Fixes#1492
This renames the old `emoji_dump.py` to `build_emoji`, removing the
old shell essentially empty shell script. `emoji_dump.py` was always
a weird name, and this makes it a bit easier to read the code for this
system.
The old "zulip_internal" name was from back when Zulip, Inc. had two
distributions of Zulip, the enterprise distribution in puppet/zulip/
and the "internal" SAAS distribution in puppet/zulip_internal. I
think the name is a bit confusing in the new fully open-source Zulip
work, so we're replacing it with "zulip_ops". I don't think the new
name is perfect, but it's better.
In the following commits, we'll delete a bunch of pieces of Zulip,
Inc.'s infrastructure that don't exist anymore and thus are no longer
useful (e.g. the old Trac configuration), with the goal of cleaning
the repository of as much unnecessary content as possible.
Previously, we checked scripts in a separate run to work around mypy
not supporting multiple scripts with the same name. Since we have
fixed that issue, we can restore the original behavior.
We leave the --scripts-only option available, though I'm not sure it's
particularly useful and we'll probably eventually remove it.
Minor fixes that enable the ability to:
- Re-run fetch-rebase-pull-request.
- Specify the name of the remote repo as an optional second parameter.
The default remains 'upstream'.
This saves a bunch of time building release tarballs, provisioning,
and upgrading Zulip from git that was spent regenerating the Zulip
emoji sprite sheet.
[commit message tweaked by tabbott]
(Most of this work was done by acrefoot in an earlier branch.
I took over the branch to fix casper tests that were broken during
the upgrade (which were fixed in a different commit). I also
made most of the changes to run-casper.)
This also upgrades phantomjs to 2.1.7.
The huge structural change here is that we no longer vendor casperjs
or download phantomjs with our own script. Instead, we just use
casperjs and phantomjs from npm, via package.json.
Another thing that we do now is run casperjs tests individually, so
that we don't get strange test flakes from test interactions. (Tests
can still influence each other in terms of changing data, since we
don't yet have code to clear the test database in between tests.)
A lot of this diff is just removing files and obsolete configurations.
The main new piece is in package.json, which causes npm to install the
new version.
Also, run-casper now runs files individually, as mentioned above.
We had vendored casperjs in the past. I didn't bring over any of our
changes. Some of the changes were performance-related (primarily
5fd58cf249), so the upgraded version may
be slower in some instances. (I didn't do much measurement of that,
since most of our slowness when running tests is about the setup
environment, not casper itself.) Any bug fixes that we may have
implemented in the past were either magically fixed by changes to
casper itself or by improvements we have made in the tests themselves
over the years.
Tim tested the Casper suite on his machine and running the full Casper
test suite is faster than it was before this change (1m30 vs. 1m50),
so we're at least not regressing overall performance.
Previously, running `tools/test-backend analytics/` (or any other test suite
name ending with a '/') would give a cryptic error about modules not
importing properly. This commit rstrip's the trailing slash from test suite
names given on the command line.
When running tools/provision.py in python3 mode, we used to create
a python2 venv called zulip-py2-twisted-venv. This was needed because
Zulip couldn't run tools/run-dev.py in python3. So we switched to
this virtualenv when running tools/run-dev.py.
Now that Zulip can run tools/run-dev.py in python3, we don't need
to create this virtualenv anymore.
- All necessary strings was converted to bytestring
- Added twisted as py3 dependency
- Change type annotation for method getchild of class Resource
- Remove activating python2 env section from run-dev.py script
Fixes#1256
generate-secrets.py now requires --development for development environment
setup or --production for production environment setup (and one of these
options is mandatory).
This solves the problem that it was somewhat easy to accidentally run
generate-secrets.py without the `-d` option while doing manual development
environment setup.
Fixes: #1911.
The new messages make it more obvious which services are started
from run-dev.py, and explicitly call out where to access the web
proxy to reach the Zulip web UI. This is a common confusion for
new administrators/developers. Messages are output before the
processes are launched, as run-dev.py does not currently have a
way to know if they started successfully.
Example output:
Starting Zulip services on ports: web proxy: 9991, Django: 9992, Tornado: 9993, webpack: 9994
Note: only port 9991 is exposed to the host in a Vagrant environment.
Alternate behavior for automated testing:
If run-dev.py is invoked with --test, don't include the webpack
port as it isn't used.
Tested on Ubuntu 14.04, by running run-dev.py at a shell prompt and
via the test-all script.
Fixes#1861
Previously, the generate-fixtures shell script by called into Django
multiple times in order to check whether the database was in a
reasonable state. Since there's a lot of overhead to starting up
Django, this resulted in `test-backend` and `test-js-with-casper`
being quite slow to run a single small test (2.8s or so) even on my
very fast laptop.
We fix this is by moving the checks into a new Python library, so that
we can avoid paying the Django startup overhead 3 times unnecessarily.
The result saves about 1.2s (~40%) from the time required to run a
single backend test.
Fixes#1221.
This is a first pass at building a framework for collecting various
stats about realms, users, streams, etc. Includes:
* New analytics tables for storing counts data
* Raw SQL queries for pulling data from zerver/models.py tables
* Aggregation functions for aggregating hourly stats into daily stats, and
aggregating user/stream level stats into realm level stats
* A management command for pulling the data
Note that counts.py was added to the linter exclude list due to errors
around %%s.
This optimizes the process of running individual or small groups of
backend tests (./tools/test-backend
zerver.tests.test_bugdown.FencedBlockPreprocessorTest.test_simple_quoting)
to allow the following syntaxes:
./tools/test-backend zerver/tests/test_bugdown.py
./tools/test-backend zerver.tests.test_bugdown.py
./tools/test-backend zerver/tests/test_bugdown
./tools/test-backend zerver.tests.test_bugdown
./tools/test-backend test_bugdown.py
./tools/test-backend test_bugdown
./tools/test-backend FencedBlockPreprocessorTest
./tools/test-backend FencedBlockPreprocessorTest.test_simple_quoting
Fixes#1670.
A bug in the node_cache.py code resulted in the node_modules symlink
in Zulip development environments being incorrectly owned by root.
This causes that bug to be fixed the next time a user provisions.
Probably most properly we should check for any number of spaces that
isn't 4, but that's a bit more work to do with our linter framework,
and in practice basically every CSS whitespace error we see is 2-space.
This adds an event listener (by way of delegation) to the
.message_inline_image elements that pops up the overlay and hides it
when the overlay exit is clicked.
Fixes#654.
NVM takes a specific node version and installs the node package and
a corresponding compatible npm package.
We use it in a somewhat hackish way to install node/npm globally with
a pinned version, since that's how we actually want to consume node in
our development environment.
Other details:
- Travis CI now is configured to use the version of node installed by
provision; the easiest way to do this was to sabotage the existing node
installation.
- jsdom is upgraded to a current version, which both requires recent
node and also is required for the tests to pass with recent node.
This fixes running the node tests on Xenial.
Fixes#1498.
[tweaked by tabbott]
This commit extracts compose_views() from update_subscriptions_backend(),
and it implements the correct behavior for forcing transactions to roll
back, which is to raise an exception.
There were really three steps in this commit:
- Extract buggy code to compose_views().
- Add tests on compose_views().
- Fix bugs exposed by the new tests by converting errors to exceptions.
We're now at the point where 100% of functions checked by mypy is
fully annotated; to avoid regressions, we're enforcing the requirement
that it stay this way. We still have a moderate amount of code that
is neither checked by mypy nor annotated, but it seems reasonable to
annotate that code at the same time as we get a chance to fix the mypy
issues in it.
This is implemented by using the --disallow-untyped-defs option in
mypy by default.
Because of some recent changes to the tokenizer, we no longer
need to call is_special_html_tag() to filter out special tags.
I also tried to make the start/end logic for pushing/popping
the stack more obvious.
This code is not directly related to the template parser, so it
can safely live in its own file.
The only significant change to the code is to the signature of
`html_branches` so that it can be called without requiring a file.
Since it's only used in html_grep, that has been updated to reflect
this change.
Fixes: #1774.
This hasn't been used since before Zulip was open source, and isn't
super reusable, so we can remove it. It'll always be there in the
history if someone ends up wanting it.
While we're at it, we remove the GitPython dependency (only used for
this tool) and the example MSMTP config for the review tool.
I don't think the black/white fallback code is actually used by
default, and thus when it is used, it's usually a sign that something
is broken. We should throw an error, but at the very least it makes
sense to print a warning.
In d583710f7c, I apparently broke the
color emoji handling, which was masked (for test purposes) by the fact
that we catch an expection if color doesn't work and in that case fall
back to black and white emoji.
This adds support for using VMWare Fusion as the Vagrant provider,
which has better performance than Virtualbox at the price of being
nonfree (in all senses of the term).
We haven't done solid benchmarking as to how much faster it is than
the Virtualbox provider.
Fixed IndexError when there is only zero or more whitespace characters
between < and >. (str.split() will return an empty list in this case, which
means there is no index 0.)
* Replace generic Exception with TemplateParserException.
* Add tests to cover many of the uncovered lines in
tools/lib/template_parser.py.
* Add an exclusion line to the naïve pattern for checking for missing
tuples in format strings, to keep the linter happy.
Tim Abbott
Tim Abbott
AZtheAsian
ausDensk
Steve Howell
Jackson
Rafid Aslam
anirudhjain75
Sampriti Panda
K.Kanakhin
Tommy Ip
lonerz
Rishi Gupta
Arpith Siromoney
Tomasz Kolek
Umair Khan
reallyly
Adarsh S
Brock Whittaker
Zac Pullar-Strecker
Tudor Nazarie
Joy Chen
Jason Le
bulat22101
kevv87
Sidhant Bhavnani
Alex Huang
Vamshi Balanaga
Bickio
Heidi Dong
Anders Kaseorg
sylvan1
Vladislav Manchev
paxapy
Pweaver (Paul Weaver)
Christie Koehler
Shashank
Tomasz Kolek
sonali0901
umkay
deekshaarul
Prabhakar Gupta
Sumana Harihareswara
Eklavya Sharma
Rishi Gupta
Diptanshu8
Feorlen
Sahil Dua
Gordon P. Hemsley