Commit Graph

2611 Commits

Author SHA1 Message Date
Eeshan Garg 929724e5e7 api docs: Add page for common error payloads.
We now have a separate page for common error payloads, for example,
the payload for when the client's API key is invalid. All error
payloads that are presented on this page will be tested similarly
to our other non-error sample fixtures.
2018-02-08 17:58:41 -08:00
Greg Price 86590dfdbe test-install: Add command destroy-all to clean up test containers.
This is just the one-liner I've been keeping in my shell history,
cleaned up a bit (newlines!) and with 28 lines of CLI boilerplate
added in front.
2018-02-08 17:29:41 -08:00
Greg Price 6e633f8e2f install: Use readlink -f rather than realpath.
It does exactly the same thing, though the name is less transparent; and
it simplifies the script by avoiding an extra, early `apt-get install`.
2018-02-08 17:22:02 -08:00
Greg Price 4c5326ce85 test-install: Factor out booted-yet-p polling loop, use in prepare-base.
Otherwise prepare-base is likely to fail when first run (but then
succeed when rerun, because the container is left running), because
the container isn't up yet when we try to operate in it.

Also clean up the placement of `set -e` vs `set -x`.
2018-02-08 16:34:49 -08:00
Greg Price fc9970e561 test-install: Add xenial support. 2018-02-08 16:34:49 -08:00
Tim Abbott 9249454518 update-authors-json: Fix handling deleted GitHub accounts.
Apparently, we've now had the first time one of our contributors had
their account deleted (at least, the author page for the contributor
who has 21 commits in python-zulip-api now 404s).
2018-02-08 12:17:20 -08:00
Umair Khan 764cd977d6 test-queue-worker-reload: Increase delay to 1.3s.
See 625939 for more information. In short, the purpose of this delay is
to give autoreload code enough time to touch every watched file at least
once before the change is made.
2018-02-06 10:35:20 -08:00
Umair Khan c415cc74d7 validate_stream_message_address_info: Add i18n tags.
Fixes #7076
2018-02-05 16:26:24 -08:00
rht 396a8b644d provision.py: Colorize error messages for fail condition. 2018-02-05 11:44:33 -08:00
Tim Abbott 0c4aa48895 provision: Clean up error message for missing virtualenv.
This new message should be clearer; it does come up occasionally.
2018-02-05 11:44:33 -08:00
rht e8a5f68980 provision: Extract color codes into standard variables.
This uses the standard color codes we have in the Python
`zulip_tools` library.

Tweaked by tabbott to clean up some bugs.
2018-02-05 11:44:33 -08:00
Greg Price e3c82fa3ec tools/test-all: Fix small desync with CI scripts.
This script and our CI scripts tools/travis/{backend,frontend} have
stayed pretty well in sync in the 6 months since 360c27ded made that
relationship explicit and easy to check!

Just one small exception; so fix that.
2018-02-01 11:38:51 -08:00
Greg Price 34f4c822cb ci: Cut out commented-out versions of tools/lint calls.
This may or may not be temporary, but either way, the other code is
there in source control, and the "why" of disabling gitlint is the
helpful bit for a comment.
2018-02-01 11:37:00 -08:00
Robert Hönig 53b4c793f6 setup_venvs.py: Enforce Python 2.7 for zulip-thumbor-venv setup.
zulip-thumbor-venv runs only with Python 2.7. This could cause
provision failures on setups where Python 2.7 is not the default.
2018-01-31 11:29:34 -08:00
Greg Price f995ad1ed5 circleci: Simplify Dockerfile templating a bit.
Injecting the generated-file warning into the settings dict felt a
little unnecessarily magical.  A warning like this is always going
to be at the top; the way it might differ between files is mainly
if the syntax for a comment varies, and in that case a simple
substitution like we're doing in this template wouldn't be enough
to express the difference anyway.  So, embrace the hardcoding.

Now, the template and the images.yml entry have a very simple
relationship: the keys in one are exactly the keys in the other.
That's good for people quickly and confidently understanding it.
2018-01-31 10:53:13 -08:00
Greg Price 8a0b5a29b9 circleci: Use the joy of `os.makedirs(..., exist_ok=True)`.
Since Python 3.2, we no longer need to write this little wrapper
all over our own code!  There was much rejoicing.
2018-01-31 10:53:13 -08:00
Vishnu Ks 91506f1bb3 circleci: Generate xenial dockerfiles along with trusty. 2018-01-31 10:53:13 -08:00
Vishnu Ks be328b2c7b circleci: Create script for generating Dockerfiles.
[greg: updated Dockerfile comment]
2018-01-31 10:53:13 -08:00
Eeshan Garg c158869096 Rename tools/lib/api_tests.py to zerver/lib/api_test_helpers.py.
Now that the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples depended on code in the
tools/lib/* directory, it caused the production tests to fail since
the tools/ directory wouldn't exist in a production environment.
2018-01-31 07:30:54 -05:00
Eeshan Garg a1a69a0ac2 api/render-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_example to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Greg Price 8147897ac5 test-run-dev: Clean up some basic Python style. 2018-01-30 16:08:43 -08:00
Greg Price ff8e588340 test-run-dev: Delete commented-out code.
We don't disable code by commenting it out -- that leaves a mess.
We delete it.  Remembering what the code was is what source control
is for.

This fixes cd849bc3f "test-run-dev: Disable Nagios check."
from a few weeks ago.
2018-01-30 15:57:58 -08:00
Brock Whittaker 3b600d5591 /billing/: Rename "payment.html" => "billing.html".
This matches the URL path /billing/ to the filename "billing.html".
2018-01-29 17:24:28 -08:00
Aditya Bansal 0fcf0c5052 thumbor: Add thumbor on port 9995 in development.
For now, this does nothing in a production environment, but it should
simplify the process of doing testing on the Thumbor implementation,
by integrating a lot of dependency management logic.
2018-01-29 13:10:29 -08:00
Umair Khan 8700e8aeaf i18n: Retrieve mobile translations.
This commit also adds a tool to push translation sources to Transifex.
This tool makes sure that we don't push mobile source file. Mobile
source file is supposed to be handled from Zulip-Mobile repo.
2018-01-29 12:49:51 -08:00
Greg Price 9476a3a334 test-install: Give the host a direct view of the guest's /tmp/src/.
(This is a small fixup to the main change, which was accidentally
included in a previous commit:
  08bbd7e61 "settings: Slightly simplify EMAIL_BACKEND logic."
Oops.  See there for most of the changes described here.)

The installer works out of a release-tarball tree.  We typically want
to share this tree between successive test-install runs (with an rsync
or similar command to update source files of interest) because
rebuilding a release tree from scratch is slow.  But the installer
will munge the tree; so instead of directly bind-mounting the tree
into the container, we need to give it an overlay over the tree, as a
sandbox to play in.

Previously we used lxc-copy's `-m overlay=...` feature to do this,
mounting an overlay in the container.  But then sometimes in
development we want to reach in and edit some code in the tree,
e.g. before rerunning the installer after something failed.  Reaching
inside the container for this is a pain (`ssh` would add latency, and
I haven't installed sshd in the containers; and getting rsync to work
with `lxc-attach` was beyond what I could figure out in a few minutes
of fiddling); and editing the base tree often doesn't work.

So, create the overlay with our own `mount -t overlay`, and have
`lxc-copy` just bind-mount that in.  Now the host has direct access to
the same overlay which the guest is working from.

Also this makes it past time to help the user out in finding the fresh
names we've created: first the container, now this shared tree.  Print
those at the end, rather than make the user scroll to the top and find
the right `set -x` line to copy-paste from.
2018-01-29 10:27:11 -08:00
Aditya Bansal c714235922 keyboard_shortcuts.html: Clean up to use 4 space indentation. 2018-01-27 16:06:36 -08:00
Aditya Bansal fbd0e190d0 zerver/navbar.html: Clean up to use 4 space indentation. 2018-01-27 15:26:51 -08:00
Aditya Bansal 52c40668bc compose.html: Clean up to use 4 space indentation. 2018-01-27 15:25:40 -08:00
Tim Abbott 87beacd015 check-templates: Update comments on linter. 2018-01-27 15:17:28 -08:00
Aditya Bansal bc7614645d zerver/login.html: Clean up to use 4 space and valid indentation.
This is the last commit in the series of commits for completing the
project of cleaning up our html templates to have 4 space and
valid indentation.

Fixes: #1236.
2018-01-27 15:12:23 -08:00
Aditya Bansal 0435e1d05d zerver/index.html: Clean up to use 4 space indentation.
In this commit we also fix a test which would fail as a result of
doing this cleanup since the test wasn't designed to take into
account the space chars which might occur in the beginning of a
html line.
2018-01-27 15:12:11 -08:00
Aditya Bansal 33a150b1dd hello.html: Clean up to use 4 space indentation. 2018-01-27 15:12:04 -08:00
Aditya Bansal 642e27d2b7 analytics/stats.html: Clean up to use 4 space indentation. 2018-01-27 23:06:59 +05:30
Aditya Bansal 16c40ea023 analytics/activity.html: Clean up to use 4 space indentation. 2018-01-27 23:06:58 +05:30
Aditya Bansal a5c0f286ec static/html/5xx.html: Clean up to use 4 space indentation. 2018-01-27 23:06:58 +05:30
Aditya Bansal b083c894a8 check-templates: Remove obsolete excludes.
In this commit we remove the excludes for api.html and
api_endpoints.html because those files no longer exist.
2018-01-27 23:06:58 +05:30
Greg Price 73f682ad7a Slack importer: Disable often-breaking test in CI.
This test randomly fails far too often in Travis -- I think more than
all our other tests combined.  It needs to be fixed before we can ask
everyone to look at build failures it causes.
2018-01-26 15:42:41 -08:00
Greg Price 644579c368 docs: Remove our forked RTD layout template.
Now that we have `eval_rst` and can explicitly exclude pages from the
toctree completely, we no longer need to set `includehidden`, and we
can return to using upstream's template.

(Meanwhile, our feature request upstream was successful!  See
rtfd/sphinx_rtd_theme#485, which upstream implemented just a week
after we requested it.  So that would have been another option.)

This reverts commit 11b8b8f48 "docs: Add rtd layout template."
2018-01-24 14:37:17 -08:00
Greg Price 08bbd7e61d settings: Slightly simplify EMAIL_BACKEND logic.
DEVELOPMENT is defined as just `not PRODUCTION`, but this code made
it look like things might be more complicated than that.
2018-01-24 14:34:30 -08:00
Tim Abbott 886d1028d9 check-templates: Remove obsolete exclude for api.html.
That file not longer exists.
2018-01-24 11:08:54 -08:00
Greg Price 341279a117 travis/production: Update expected HTTP output.
This changed with the recent consolidation of how we get a
self-signed cert in test installs.
2018-01-24 10:58:44 -08:00
Greg Price 841a5f3152 install: Say --self-signed-cert instead of --snakeoil-cert.
Less evocative, but requires less explanation to document because
it's a well-known term on the Internet.
2018-01-23 18:08:52 -08:00
Greg Price f4d5ade9d8 docs: Exclude Sphinx config file from mypy checking.
It runs in kind of a peculiar environment -- in particular with the
`tags` identifier injected into the namespace -- and it contains
very little code more complex than `foo = "bar"`, so there's not
much to check anyway.
2018-01-23 14:28:56 -08:00
Greg Price a727f22a4f lint: Suppress complaint about `tags` in Sphinx config. 2018-01-23 13:56:21 -08:00
Greg Price c2b77c5719 lint pyflakes: Pull out our error-suppression patterns as data.
This makes the list much cleaner to understand and edit.
2018-01-23 13:56:21 -08:00
Rhea Parekh 5b1e9f8181 slack importer: correct the implementation of unzipping slack data file.
Slack zip file unzips in the same folder.
2018-01-23 10:01:15 -05:00
Greg Price ff29fe07be provision: Install virtualenv on xenial too.
As the commit message on 680381c9d which added this for stretch says,
we need this dependency when on xenial as well.  Add it there.
2018-01-22 19:36:52 -08:00
Greg Price 2a59b2d2ac install: Work around a bug in the (our) Debian package for camo.
Before this fix, the installer has an extremely annoying bug where
when run inside a container with `lxc-attach`, when the installer
finishes, the `lxc-attach` just hangs and doesn't respond even to
C-c or C-z.  The only way to get the terminal back is to root around
from some other terminal to find the PID and kill it; then run
something like `stty sane` to fix the messed-up terminal settings
left behind.

After bisecting pieces of the install script to locate which step
was causing the issue, it comes down to the `service camo restart`.
The comment here indicates that we knew about an annoying bug here
years ago, and just swept it under the rug by skipping this step
when in Travis. >_<

The issue can be reproduced by running simply `service camo restart`
under `lxc-attach` instead of the installer; or `service camo start`,
following a `service camo stop`.  If `lxc-attach` is used to get an
interactive shell, these commands appear to work fine; but then when
that shell exits, the same hang appears.  So, when we start camo
we're evidently leaving some kind of mess that entangles the daemon
with our shell.

Looking at the camo initscript where it starts the daemon, there's
not much code, and one flag jumps out as suspicious:

  start-stop-daemon --start --quiet --pidfile $PIDFILE -bm \
    --exec $DAEMON --no-close -c nobody --test > /dev/null 2>&1 \
    || return 1
  start-stop-daemon --start --quiet --pidfile $PIDFILE -bm \
    --no-close -c nobody --exec $DAEMON -- \
    $DAEMON_ARGS >> /var/log/camo/camo.log 2>&1 \
    || return 2

What does `--no-close` do?

 -C, --no-close
     Do not close any file descriptor when forcing the daemon
     into  the  background  (since version 1.16.5).  Used for
     debugging purposes to see  the  process  output,  or  to
     redirect  file  descriptors  to  log the process output.

And in fact, looking in /proc/PID/fd while a hang is happening finds
that fd 0 on the camo daemon process, aka stdin, is connected to our
terminal.

So, stop that by denying the initscript our stdin in the first place.
This fixes the problem.

The Debian maintainer turns out to be "Zulip Debian Packaging Team",
at debian@zulip.com; so this package and its bugs are basically ours.
2018-01-22 18:55:46 -08:00
Greg Price 6e7ae9a239 test-install: Run installer under eatmydata.
This is a tool that throws away `fsync` calls and other requests for
the system to sync files to disk.  It may make the install faster; for
example, if it has to install a number of system packages, `dpkg` is
known to make a lot of `fsync` calls which slow things down
significantly.  Conversely, if there's a power failure in the middle
of running a test install, we really don't mind if the test install's
data becomes corrupt.
2018-01-22 18:55:46 -08:00