Commit Graph

2902 Commits

Author SHA1 Message Date
Greg Price 8111848ac4 test-install: Do a dist-upgrade in prepare-base.
This keeps the base tree up to date, saving the time we'd spend
doing the same upgrades in each test install.
2018-03-06 19:43:02 -08:00
Tim Abbott f253f96211 lint: Add "Gravatar" to capitalized terms in linter. 2018-03-05 10:29:18 -08:00
Tim Abbott 4d8e18e58f settings: Move display_checkmark to settings_ui.js. 2018-03-04 18:26:27 -08:00
Tim Abbott 6a06e453c7 display_settings: Extend de-duplication of unnecessary logic.
We add a new option to the change_display_setting function to support
cases with a custom on-success or on-failure message.
2018-03-04 18:09:46 -08:00
Balaji2198 5aa7098c81 display_settings: Change success/failure feedback interface.
This replaces the cumbersome system we had for giving users feedback
on settings state changes in the display settings UI.

We expect this new system to be what we will attempt to migrate other
settings widgets to match over the coming weeks and months.  It also
provides the opportunity to significant refactor away a lot of the
code duplication in settings_display.js.

Thanks to Brock Whittaker for redoing the styling and improving the
code simplicity.

Fixes #7622.
2018-03-04 17:47:05 -08:00
Tim Abbott a3ca7ee1c7 lint: Add a few strings to capitalization linter.
My hope is that we'll be able to eliminate these strings from the UI
altogether soon, but this is needed to keep CI passing for now.
2018-03-04 16:34:00 -08:00
Robert Hönig 81ba7a1e40 Mark DigestWorker and PushNotificationsWorker as nocoverage.
These two classes are tricky to test, and nocoverage-ing them
allows us to mark queue_processors.py as fully covered. We
still want to cover these two workers at some point, but for
now, it's nice to enforce full coverage for any future changes
to queue_processors.py.

Fixes (sort of) #6542.
2018-03-04 13:31:33 -08:00
Tim Abbott 530e098916 node: Remove 100% coverage requirement for user groups.
This is a temporary change to get CI passing again; we lost some
coverage in aed8cc9062, which fixed some
major bugs in the feature.
2018-03-04 10:11:11 -08:00
YJDave f5a24efdd8 static/js: Add `settings_ui.js` file for common settings UI functions. 2018-03-03 08:38:54 -08:00
Eeshan Garg 59581f6a32 tools/lint: Require a space after every `:` in JSON files. 2018-03-02 14:10:53 -08:00
Tim Abbott 335c383bbb lint: Add some excludes to capitalization linter. 2018-03-02 11:55:46 -08:00
Tim Abbott 895b5642f1 commit-message-list: Don't print anything if no commits.
This fixes some output spam when running the linter at
upstream/master.
2018-03-02 11:51:08 -08:00
Tim Abbott 14764d015a settings: Tag cookie bot placeholder short name for translation. 2018-03-02 09:33:16 -08:00
Tim Abbott e18537d842 emoji: Add a translation tag for new_emoji placeholder. 2018-03-02 09:32:09 -08:00
Tim Abbott 9c4b0cc100 lint: Clean up translation tag lint for filter settings. 2018-03-02 09:30:42 -08:00
Tim Abbott f9210bace8 lint: Add a translation tag for marketing. 2018-03-02 09:28:57 -08:00
Joshua Pan 5777582013 lint: Tighten placeholder lint rule to exclude links.
Fixes #8420.
2018-03-02 09:24:09 -08:00
Tim Abbott 5ccd3aa71b i18n: Run process-mobile-i18n when syncing translations. 2018-03-02 08:59:00 -08:00
Tim Abbott 3be0b3dd91 mypy: Fix errors in linter.
We should figure out a better way to manage this issue.
2018-03-01 20:33:30 -08:00
Tim Abbott 8d00266fd6 lint: Add a linter rule banning use of sudo in most scripts.
This should help prevent against bugs where we accidentally introduce
use of sudo somewhere in the production installer or upgrade code path
(these used to happen all the time), which doesn't work on production
systems that don't have sudo setup.
2018-03-01 16:30:31 -08:00
Greg Price 72497d61c1 test-migrations: Clean up a bit. 2018-03-01 15:03:55 -08:00
Aditya Bansal bd0a5d5d69 css-minifier: Lint for '+' operator in calc().
Fixes: #8403.
2018-03-01 13:26:16 -08:00
Tim Abbott 6b31752f82 user_groups: Change placeholder group to "Marketing team".
This is much more something that would apply to likely customer
organizations.
2018-03-01 11:28:24 -08:00
Vishnu Ks 36f6ad76b4 admin ui: Set usergroup name input maxlength to 100.
Same as that of the maxlength defined in backend.
2018-03-01 11:27:26 -08:00
Greg Price 4475950ddf queue: Restore prematurely-cut upgrade path.
Revert c8f034e9a "queue: Remove missedmessage_email_senders code."
As the comment in the code says, it ensures a smooth upgrade path
from 1.7.x; we can delete it in master after 1.8.0 is released.
The removal commit was merged early due to a communication failure.
2018-02-28 11:15:53 -08:00
Umair Khan c8f034e9a0 queue: Remove missedmessage_email_senders code.
After 68513952fb, all emails are sent through email_senders queue.
This commit removes code related to the legacy queue.
2018-02-21 16:43:56 -08:00
Eeshan Garg be0a04f33e api docs: Test sample fixtures for unauthorized_errors_fatal.
This commit adds tests for the sample fixtures for when
unauthorized_errors_fatal is passed to client.add_subscriptions.
2018-02-20 14:54:12 -08:00
Eeshan Garg 06bf47d2f5 api docs: Test sample fixture for user not authorized error.
This commit adds tests for the fixture for when a user is not
authorized (perhaps because the query requires the use of admin
privileges) for a particular query.
2018-02-20 14:52:09 -08:00
Eeshan Garg 124c672a7c api docs: Test sample fixture for message edit permission error.
In templates/zerver/api/update-message.md, we have a sample fixture
for when a zulip.Client does not have the permission to update/edit
a particular message. This commit adds a test for that fixture.
Also, tools/test-api now also uses a non-admin client for this test,
which might come in handy in the future.
2018-02-20 14:52:09 -08:00
Steve Howell 1f6ddf0110 refactor: Extract transmit.js from compose.js.
We now isolate the code to transmit messages into transmit.js.
It is stable code that most folks doing UI work in compose.js don't
care about the details of, so it's just clutter there.  Also, we may
soon have other widgets than the compose box that send messages.

This change mostly preserves test coverage, although in some cases
we stub at a higher level for the compose path (this is a good thing).
Extracting out transmit.js allows us to lock down 100% coverage on that
file.
2018-02-20 09:29:26 -08:00
rht 70a41cc2eb tools/tests: Use Python 3 syntax for typing. 2018-02-19 10:17:03 -08:00
ihsavru 4103996a22 org-settings: Fix styling of Allowed Domains modal.
Fixes: #7628.

Cleaned up by Brock Whittaker and Node tests fixed by Steve Howell.
2018-02-16 16:00:21 -08:00
Aditya Bansal 6fce1d7834 template_parser: Add parsing support for self closing tags as per HTML5.
In this commit we add support for some tags which are also called
void-elements according to
http://w3c.github.io/html/syntax.html#void-elements to be parsed by
our template parser and get tagged as singleton_html_tags.

Fixes: #8387.
2018-02-15 17:47:43 -05:00
Greg Price b0b0777cd0 translation: Make new tagmessages script a bit easier to run.
Now executable! Just run `tools/tagmessages`.

Also, get the username and password from a `.transifexrc` file.

And hardcode the project slug to `zulip-test` rather than to `zulip`;
the Transifex API is bad at namespacing, so this makes it possible to
run this script on a test project (the only way we're currently using
it) even for people like me who can also upload to the real Zulip
project on Transifex.
2018-02-15 13:38:09 -08:00
Robert Hönig ebf79427ca translation: Add script to tag strings in Transifex.
For now, tags are only used for backend translations with django.
2018-02-15 13:38:09 -08:00
Tim Abbott be3d43bb46 lint: Add linter rule banning $.get and friends.
These are not allowed in our style guide.
2018-02-13 16:47:59 -08:00
Shubham Padia e1f943913a lint: Allow revert commit messages in gitlint.
Fixes #8145.
2018-02-13 09:21:01 -08:00
Aditya Bansal 35969edd66 deps: Replace libz-dev with zlib1g-dev since the former was renamed. 2018-02-12 14:40:26 -08:00
Aditya Bansal d5736ef65a provision.py: Remove duplicate install candidated from APT_DEPENDENCIES.
This is necessary, since we now have some dependencies in common
between the Thumbor and main Zulip dependency lists.
2018-02-12 14:40:26 -08:00
Rhea Parekh 3ebd30120c slack importer: Remove depreciated test.
This test was added in '/tests/test_slack_importer' in
6addf79edb.
2018-02-12 00:43:02 +05:30
Greg Price b374ea7357 mypy: Pass --show-traceback.
This saves a bit of debugging in the event that we hit a crash
inside mypy.
2018-02-09 19:42:49 -08:00
Eeshan Garg 929724e5e7 api docs: Add page for common error payloads.
We now have a separate page for common error payloads, for example,
the payload for when the client's API key is invalid. All error
payloads that are presented on this page will be tested similarly
to our other non-error sample fixtures.
2018-02-08 17:58:41 -08:00
Greg Price 86590dfdbe test-install: Add command destroy-all to clean up test containers.
This is just the one-liner I've been keeping in my shell history,
cleaned up a bit (newlines!) and with 28 lines of CLI boilerplate
added in front.
2018-02-08 17:29:41 -08:00
Greg Price 6e633f8e2f install: Use readlink -f rather than realpath.
It does exactly the same thing, though the name is less transparent; and
it simplifies the script by avoiding an extra, early `apt-get install`.
2018-02-08 17:22:02 -08:00
Greg Price 4c5326ce85 test-install: Factor out booted-yet-p polling loop, use in prepare-base.
Otherwise prepare-base is likely to fail when first run (but then
succeed when rerun, because the container is left running), because
the container isn't up yet when we try to operate in it.

Also clean up the placement of `set -e` vs `set -x`.
2018-02-08 16:34:49 -08:00
Greg Price fc9970e561 test-install: Add xenial support. 2018-02-08 16:34:49 -08:00
Tim Abbott 9249454518 update-authors-json: Fix handling deleted GitHub accounts.
Apparently, we've now had the first time one of our contributors had
their account deleted (at least, the author page for the contributor
who has 21 commits in python-zulip-api now 404s).
2018-02-08 12:17:20 -08:00
Umair Khan 764cd977d6 test-queue-worker-reload: Increase delay to 1.3s.
See 625939 for more information. In short, the purpose of this delay is
to give autoreload code enough time to touch every watched file at least
once before the change is made.
2018-02-06 10:35:20 -08:00
Umair Khan c415cc74d7 validate_stream_message_address_info: Add i18n tags.
Fixes #7076
2018-02-05 16:26:24 -08:00
rht 396a8b644d provision.py: Colorize error messages for fail condition. 2018-02-05 11:44:33 -08:00
Tim Abbott 0c4aa48895 provision: Clean up error message for missing virtualenv.
This new message should be clearer; it does come up occasionally.
2018-02-05 11:44:33 -08:00
rht e8a5f68980 provision: Extract color codes into standard variables.
This uses the standard color codes we have in the Python
`zulip_tools` library.

Tweaked by tabbott to clean up some bugs.
2018-02-05 11:44:33 -08:00
Greg Price e3c82fa3ec tools/test-all: Fix small desync with CI scripts.
This script and our CI scripts tools/travis/{backend,frontend} have
stayed pretty well in sync in the 6 months since 360c27ded made that
relationship explicit and easy to check!

Just one small exception; so fix that.
2018-02-01 11:38:51 -08:00
Greg Price 34f4c822cb ci: Cut out commented-out versions of tools/lint calls.
This may or may not be temporary, but either way, the other code is
there in source control, and the "why" of disabling gitlint is the
helpful bit for a comment.
2018-02-01 11:37:00 -08:00
Robert Hönig 53b4c793f6 setup_venvs.py: Enforce Python 2.7 for zulip-thumbor-venv setup.
zulip-thumbor-venv runs only with Python 2.7. This could cause
provision failures on setups where Python 2.7 is not the default.
2018-01-31 11:29:34 -08:00
Greg Price f995ad1ed5 circleci: Simplify Dockerfile templating a bit.
Injecting the generated-file warning into the settings dict felt a
little unnecessarily magical.  A warning like this is always going
to be at the top; the way it might differ between files is mainly
if the syntax for a comment varies, and in that case a simple
substitution like we're doing in this template wouldn't be enough
to express the difference anyway.  So, embrace the hardcoding.

Now, the template and the images.yml entry have a very simple
relationship: the keys in one are exactly the keys in the other.
That's good for people quickly and confidently understanding it.
2018-01-31 10:53:13 -08:00
Greg Price 8a0b5a29b9 circleci: Use the joy of `os.makedirs(..., exist_ok=True)`.
Since Python 3.2, we no longer need to write this little wrapper
all over our own code!  There was much rejoicing.
2018-01-31 10:53:13 -08:00
Vishnu Ks 91506f1bb3 circleci: Generate xenial dockerfiles along with trusty. 2018-01-31 10:53:13 -08:00
Vishnu Ks be328b2c7b circleci: Create script for generating Dockerfiles.
[greg: updated Dockerfile comment]
2018-01-31 10:53:13 -08:00
Eeshan Garg c158869096 Rename tools/lib/api_tests.py to zerver/lib/api_test_helpers.py.
Now that the Markdown extension defined in
zerver/lib/bugdown/api_generate_examples depended on code in the
tools/lib/* directory, it caused the production tests to fail since
the tools/ directory wouldn't exist in a production environment.
2018-01-31 07:30:54 -05:00
Eeshan Garg a1a69a0ac2 api/render-message: Make code examples and fixtures testable.
This commit uses the Markdown extension defined in
zerver/lib/bugdown/api_generate_example to generate the example
fixture and code example, so that both are tested in
tools/lib/api_tests.
2018-01-31 07:30:54 -05:00
Greg Price 8147897ac5 test-run-dev: Clean up some basic Python style. 2018-01-30 16:08:43 -08:00
Greg Price ff8e588340 test-run-dev: Delete commented-out code.
We don't disable code by commenting it out -- that leaves a mess.
We delete it.  Remembering what the code was is what source control
is for.

This fixes cd849bc3f "test-run-dev: Disable Nagios check."
from a few weeks ago.
2018-01-30 15:57:58 -08:00
Brock Whittaker 3b600d5591 /billing/: Rename "payment.html" => "billing.html".
This matches the URL path /billing/ to the filename "billing.html".
2018-01-29 17:24:28 -08:00
Aditya Bansal 0fcf0c5052 thumbor: Add thumbor on port 9995 in development.
For now, this does nothing in a production environment, but it should
simplify the process of doing testing on the Thumbor implementation,
by integrating a lot of dependency management logic.
2018-01-29 13:10:29 -08:00
Umair Khan 8700e8aeaf i18n: Retrieve mobile translations.
This commit also adds a tool to push translation sources to Transifex.
This tool makes sure that we don't push mobile source file. Mobile
source file is supposed to be handled from Zulip-Mobile repo.
2018-01-29 12:49:51 -08:00
Greg Price 9476a3a334 test-install: Give the host a direct view of the guest's /tmp/src/.
(This is a small fixup to the main change, which was accidentally
included in a previous commit:
  08bbd7e61 "settings: Slightly simplify EMAIL_BACKEND logic."
Oops.  See there for most of the changes described here.)

The installer works out of a release-tarball tree.  We typically want
to share this tree between successive test-install runs (with an rsync
or similar command to update source files of interest) because
rebuilding a release tree from scratch is slow.  But the installer
will munge the tree; so instead of directly bind-mounting the tree
into the container, we need to give it an overlay over the tree, as a
sandbox to play in.

Previously we used lxc-copy's `-m overlay=...` feature to do this,
mounting an overlay in the container.  But then sometimes in
development we want to reach in and edit some code in the tree,
e.g. before rerunning the installer after something failed.  Reaching
inside the container for this is a pain (`ssh` would add latency, and
I haven't installed sshd in the containers; and getting rsync to work
with `lxc-attach` was beyond what I could figure out in a few minutes
of fiddling); and editing the base tree often doesn't work.

So, create the overlay with our own `mount -t overlay`, and have
`lxc-copy` just bind-mount that in.  Now the host has direct access to
the same overlay which the guest is working from.

Also this makes it past time to help the user out in finding the fresh
names we've created: first the container, now this shared tree.  Print
those at the end, rather than make the user scroll to the top and find
the right `set -x` line to copy-paste from.
2018-01-29 10:27:11 -08:00
Aditya Bansal c714235922 keyboard_shortcuts.html: Clean up to use 4 space indentation. 2018-01-27 16:06:36 -08:00
Aditya Bansal fbd0e190d0 zerver/navbar.html: Clean up to use 4 space indentation. 2018-01-27 15:26:51 -08:00
Aditya Bansal 52c40668bc compose.html: Clean up to use 4 space indentation. 2018-01-27 15:25:40 -08:00
Tim Abbott 87beacd015 check-templates: Update comments on linter. 2018-01-27 15:17:28 -08:00
Aditya Bansal bc7614645d zerver/login.html: Clean up to use 4 space and valid indentation.
This is the last commit in the series of commits for completing the
project of cleaning up our html templates to have 4 space and
valid indentation.

Fixes: #1236.
2018-01-27 15:12:23 -08:00
Aditya Bansal 0435e1d05d zerver/index.html: Clean up to use 4 space indentation.
In this commit we also fix a test which would fail as a result of
doing this cleanup since the test wasn't designed to take into
account the space chars which might occur in the beginning of a
html line.
2018-01-27 15:12:11 -08:00
Aditya Bansal 33a150b1dd hello.html: Clean up to use 4 space indentation. 2018-01-27 15:12:04 -08:00
Aditya Bansal 642e27d2b7 analytics/stats.html: Clean up to use 4 space indentation. 2018-01-27 23:06:59 +05:30
Aditya Bansal 16c40ea023 analytics/activity.html: Clean up to use 4 space indentation. 2018-01-27 23:06:58 +05:30
Aditya Bansal a5c0f286ec static/html/5xx.html: Clean up to use 4 space indentation. 2018-01-27 23:06:58 +05:30
Aditya Bansal b083c894a8 check-templates: Remove obsolete excludes.
In this commit we remove the excludes for api.html and
api_endpoints.html because those files no longer exist.
2018-01-27 23:06:58 +05:30
Greg Price 73f682ad7a Slack importer: Disable often-breaking test in CI.
This test randomly fails far too often in Travis -- I think more than
all our other tests combined.  It needs to be fixed before we can ask
everyone to look at build failures it causes.
2018-01-26 15:42:41 -08:00
Greg Price 644579c368 docs: Remove our forked RTD layout template.
Now that we have `eval_rst` and can explicitly exclude pages from the
toctree completely, we no longer need to set `includehidden`, and we
can return to using upstream's template.

(Meanwhile, our feature request upstream was successful!  See
rtfd/sphinx_rtd_theme#485, which upstream implemented just a week
after we requested it.  So that would have been another option.)

This reverts commit 11b8b8f48 "docs: Add rtd layout template."
2018-01-24 14:37:17 -08:00
Greg Price 08bbd7e61d settings: Slightly simplify EMAIL_BACKEND logic.
DEVELOPMENT is defined as just `not PRODUCTION`, but this code made
it look like things might be more complicated than that.
2018-01-24 14:34:30 -08:00
Tim Abbott 886d1028d9 check-templates: Remove obsolete exclude for api.html.
That file not longer exists.
2018-01-24 11:08:54 -08:00
Greg Price 341279a117 travis/production: Update expected HTTP output.
This changed with the recent consolidation of how we get a
self-signed cert in test installs.
2018-01-24 10:58:44 -08:00
Greg Price 841a5f3152 install: Say --self-signed-cert instead of --snakeoil-cert.
Less evocative, but requires less explanation to document because
it's a well-known term on the Internet.
2018-01-23 18:08:52 -08:00
Greg Price f4d5ade9d8 docs: Exclude Sphinx config file from mypy checking.
It runs in kind of a peculiar environment -- in particular with the
`tags` identifier injected into the namespace -- and it contains
very little code more complex than `foo = "bar"`, so there's not
much to check anyway.
2018-01-23 14:28:56 -08:00
Greg Price a727f22a4f lint: Suppress complaint about `tags` in Sphinx config. 2018-01-23 13:56:21 -08:00
Greg Price c2b77c5719 lint pyflakes: Pull out our error-suppression patterns as data.
This makes the list much cleaner to understand and edit.
2018-01-23 13:56:21 -08:00
Rhea Parekh 5b1e9f8181 slack importer: correct the implementation of unzipping slack data file.
Slack zip file unzips in the same folder.
2018-01-23 10:01:15 -05:00
Greg Price ff29fe07be provision: Install virtualenv on xenial too.
As the commit message on 680381c9d which added this for stretch says,
we need this dependency when on xenial as well.  Add it there.
2018-01-22 19:36:52 -08:00
Greg Price 2a59b2d2ac install: Work around a bug in the (our) Debian package for camo.
Before this fix, the installer has an extremely annoying bug where
when run inside a container with `lxc-attach`, when the installer
finishes, the `lxc-attach` just hangs and doesn't respond even to
C-c or C-z.  The only way to get the terminal back is to root around
from some other terminal to find the PID and kill it; then run
something like `stty sane` to fix the messed-up terminal settings
left behind.

After bisecting pieces of the install script to locate which step
was causing the issue, it comes down to the `service camo restart`.
The comment here indicates that we knew about an annoying bug here
years ago, and just swept it under the rug by skipping this step
when in Travis. >_<

The issue can be reproduced by running simply `service camo restart`
under `lxc-attach` instead of the installer; or `service camo start`,
following a `service camo stop`.  If `lxc-attach` is used to get an
interactive shell, these commands appear to work fine; but then when
that shell exits, the same hang appears.  So, when we start camo
we're evidently leaving some kind of mess that entangles the daemon
with our shell.

Looking at the camo initscript where it starts the daemon, there's
not much code, and one flag jumps out as suspicious:

  start-stop-daemon --start --quiet --pidfile $PIDFILE -bm \
    --exec $DAEMON --no-close -c nobody --test > /dev/null 2>&1 \
    || return 1
  start-stop-daemon --start --quiet --pidfile $PIDFILE -bm \
    --no-close -c nobody --exec $DAEMON -- \
    $DAEMON_ARGS >> /var/log/camo/camo.log 2>&1 \
    || return 2

What does `--no-close` do?

 -C, --no-close
     Do not close any file descriptor when forcing the daemon
     into  the  background  (since version 1.16.5).  Used for
     debugging purposes to see  the  process  output,  or  to
     redirect  file  descriptors  to  log the process output.

And in fact, looking in /proc/PID/fd while a hang is happening finds
that fd 0 on the camo daemon process, aka stdin, is connected to our
terminal.

So, stop that by denying the initscript our stdin in the first place.
This fixes the problem.

The Debian maintainer turns out to be "Zulip Debian Packaging Team",
at debian@zulip.com; so this package and its bugs are basically ours.
2018-01-22 18:55:46 -08:00
Greg Price 6e7ae9a239 test-install: Run installer under eatmydata.
This is a tool that throws away `fsync` calls and other requests for
the system to sync files to disk.  It may make the install faster; for
example, if it has to install a number of system packages, `dpkg` is
known to make a lot of `fsync` calls which slow things down
significantly.  Conversely, if there's a power failure in the middle
of running a test install, we really don't mind if the test install's
data becomes corrupt.
2018-01-22 18:55:46 -08:00
Greg Price eb25928674 test-install: Allow the installer to move the install tree aside.
When the install script is successful, one of the final things it
wants to do is to move the tree that Zulip was installed from into the
deployments directory.  It can't do that, at least not in a naive way
with `mv`, if the tree is actually a mount point.  So, stick the tree
inside some other directory that we create just for the purpose of
being the mount point and containing the install tree.
2018-01-22 18:55:46 -08:00
Greg Price 33a0a00705 production test suite: Use installer's own `--snakeoil-cert`. 2018-01-22 18:55:46 -08:00
Greg Price b0a9117e80 test-install: Pre-install a few more dependencies in base image. 2018-01-22 18:55:46 -08:00
Greg Price 0e3ab4d437 test-install: Share the pip cache across installs.
This saves several minutes off the install time.  Sadly pip still
clones Git repos for dependencies that point to them, but for many
others (not all? not sure) it just gets a wheel from the cache.
2018-01-22 18:55:46 -08:00
Greg Price 69ba6ad6d7 test-install: Let installer handle the snakeoil cert. 2018-01-22 18:55:46 -08:00
Greg Price 7b47cca67e test-install: Pre-install two more dependencies in base image. 2018-01-22 18:55:46 -08:00
Greg Price 525b136f10 install: Install curl.
The third-party `install-yarn.sh` script uses `curl`, and we invoke it
in `install-node`.  So we need to install it as a dependency.

We've mostly gotten away with this because it's common for `curl` to
already be installed; but it isn't always.
2018-01-22 18:55:46 -08:00
Greg Price 07969a2b0c test-install: Share the tarball directory between host and container.
This greatly simplifies iterating on changes to the installer and
associated code: just edit in the shared directory (or edit in your
worktree and rsync to the directory), and rerun.

With this change, the form with a directory is now really the main
way to run the script; the form accepting a tarball is really just
a convenience feature, unpacking the tarball and then proceeding with
that directory.
2018-01-22 18:55:46 -08:00
Greg Price d7e2190b85 test-install: Pass options through to the installer.
This will facilitate testing interesting installer features
using its own CLI.

On my laptop, with a recent base image (updated a few days ago with
`prepare-base`), it takes just 7 or 8 seconds to get to the installer
running, as timed by passing `--help` so that the installer promptly
exits.
2018-01-22 18:55:45 -08:00
Greg Price de7abd8f78 test-install: Upgrade CLI parsing, with getopt.
This will let us add more options without the CLI collapsing under
its own weight.
2018-01-22 18:55:45 -08:00
Aditya Bansal efbddce34d settings_user_groups.js: Add 100% node test coverage. 2018-01-20 08:01:06 -05:00
Greg Price bf5f1b5f20 install: Start on an LXC-based dev/test environment for the installer.
In order to do development on the installer itself in a sane way,
we need a reasonably fast and automatic way to get a fresh environment
to try to run it in.

This calls for some form of virtualization.  Choices include

 * A public cloud, like EC2 or Digital Ocean.  These could work, if we
   wrote some suitable scripts against their APIs, to manage
   appropriate base images (as AMIs or snapshots respectively) and to
   start fresh instances/droplets from a base image.  There'd be some
   latency on starting a new VM, and this would also require the user
   to have an account on the relevant cloud with API access to create
   images and VMs.

 * A local whole-machine VM system (hypervisor) like VirtualBox or
   VMware, perhaps managing the configuration through Vagrant.  These
   hypervisors can be unstable and painfully slow.  They're often the
   only way to get development work done on a Mac or Windows machine,
   which is why we use them there for the normal Zulip development
   environment; but I don't really want to find out how their
   instability scales when constantly spawning fresh VMs from an image.

 * Containers.  The new hotness, the name on everyone's lips, is Docker.
   But Docker is not designed for virtualizing a traditional Unix server,
   complete with its own init system and a fleet of processes with a
   shared filesystem -- in other words, the platform Zulip's installer
   and deployment system are for.  Docker brings its own quite
   different model of deployment, and someday we may port Zulip from
   the traditional Unix server to the Docker-style deployment model,
   but for testing our traditional-Unix-server deployment we need a
   (virtualized) traditional Unix server.

 * Containers, with LXC.  LXC provides containers that function as
   traditional Unix servers; because of the magic of containers, the
   overhead is quite low, and LXC offers handy snapshotting features
   so that we can quickly start up a fresh environment from a base
   image.  Running LXC does require a Linux base system.  For
   contributors whose local development machine isn't already Linux,
   the same solutions are available as for our normal development
   environment: the base system for running LXC could be e.g. a
   Vagrant-managed VirtualBox VM, or a machine in a public cloud.

This commit adds a first version of such a thing, using LXC to manage
a base image plus a fresh container for each test run.  The test
containers function as VMs: once installed, all the Zulip services run
normally in them and can be managed in the normal production ways.

This initial version has a shortage of usage messages or docs, and
likely has some sharp edges.  It also requires familiarity with the
basics of LXC commands in order to make good use of the resulting
containers: `lxc-ls -f`, `lxc-attach`, `lxc-stop`, and `lxc-start`,
in particular.
2018-01-19 17:27:04 -08:00
Aditya Bansal 76f6f7cb47 datetimepicker: Add flatpickr lib as dependancy. 2018-01-19 11:33:11 -05:00
Aditya Bansal c770bdaa3a reminder_bot: Add infra for adding reminder bot to every realm. 2018-01-19 11:33:11 -05:00
Vishnu Ks 0bca0286a1 billing: Integrate Stripe, using Stripe Checkout.
Stripe Checkout means using JS code provided by Stripe to handle
almost all of the UI, which is great for us.

There are more features we should add to this page and changes we
should make, but this gives us an MVP.

[greg: expanded commit message; fixed import ordering and some types.]
2018-01-17 16:43:54 -08:00
Rhea Parekh c1d336f5d1 slack importer: Refactor checking for availability of realm subdomain. 2018-01-17 09:38:53 -05:00
Steve Howell 3cb6871f90 Improve messaging in test_server_running().
We now make it a bit easier to see that you are waiting for
the server, as well as indicating when the server is up.
2018-01-16 13:25:19 -05:00
Rohitt Vashishtha cc8942e0dc /team: Fetch zulip-js repository data. 2018-01-16 12:23:19 -05:00
Aditya Bansal ec1297c1e8 schedulemessages: Add delivery system for scheduled message. 2018-01-10 09:18:02 -05:00
Greg Price 1a83b32da5 circleci: Skip checking links in documentation for now.
This is fairly often -- though not always! -- failing, with a nasty
failure mode where it takes like 6 minutes to time out.  See
discussion on #7748 (search for "bad link").

Actually, after seeing it happen just now when running
test-documentation on my laptop, on some other link, it occurs to me
that I've seen this before -- it's fairly common in Travis, too.  It's
just that it doesn't actually cause the build to fail :-/, and on
Travis we haven't been paying as close attention to slow builds as we
are on Circle right now.
2018-01-09 10:53:13 -08:00
Greg Price d66f081af8 test-documentation: Send messages to stderr, not stdout.
Generally stderr is the conventional place for this sort of running
commentary, and it's better set up for it: by default stdout may have
a buffer inside the process so that things written to it don't reach
the outside until later, while stderr is always by default unbuffered,
so messages are printed immediately.

Here, until the previous commit, because our color-reset sequence was
being printed without a following newline (with `echo -n`), it was
getting buffered; and then error messages from `scrapy` to stderr were
being erroneously painted with the color intended for the message
"Testing links in documentation...".
2018-01-09 10:15:49 -08:00
Greg Price 618beb7110 test-documentation: Factor out the terminal goo for coloring messages. 2018-01-09 10:15:08 -08:00
Greg Price f1f5b25969 test-documentation: Fix whitespace for readability. 2018-01-09 10:08:12 -08:00
Vishnu Ks ec207e5697 slack importer: Use standalone database for running tests. 2018-01-07 09:50:27 -05:00
Cynthia Lin a937750b1a node tests: Add 100% test coverage for user_groups.js. 2018-01-06 11:50:52 -05:00
Cynthia Lin 83706e1bcd org settings: Add User groups content and design. 2018-01-06 11:50:52 -05:00
Umair Khan 7145e6c97b travis: Enable test-queue-worker-reload.
Fixes #1341
2018-01-05 08:24:57 -05:00
Umair Khan 6259390f40 test-queue-worker-reload: Add wait to avoid race condition.
The autoreload code of Django works by looping over the files associated
with all the loaded modules. This loop is run after every 1 second. If
the file is found for the first time by the loop, it is assumed that the
file is new and is not modified between the time it is loaded and is
checked by the loop. This assumption is the source of a race condition.

We can either implement a more sensitive version of the loop or we can
just allow enough time to the Django loop to touch every file at least
once.

For the time being, we are going with the second option.
2018-01-05 08:24:57 -05:00
Umair Khan e68adab75d test-queue-worker-reload: Poll for 60 seconds. 2018-01-05 08:24:57 -05:00
Umair Khan 21eabf4a60 travis: Fix test-queue-worker-reload.
Previously, there were following problems with the implmentation:
* Same file handle was being used to read and write. We used to do
  `seek(0)` and then `read()`. This had a chance to overwrite
  file data. Now we use different file handles to read and write data.
* We were using text streams. Text streams cannot be used with
  `bufferring=0`. Now we use binary streams without buffering so that
  data is available for reading without any delay.

This commit also updates the key(s) that we search in the logfile.
Previously, launch of all queues was announced in the log, now we only
anounce the number of threads that were launched.

This commit also makes sure that we always exit after gracefull shutting
down the development server.
2018-01-05 08:24:57 -05:00
Steve Howell a49fd44612 css linter: Fix error reporting.
Three changes:
    * Change wording of error message.
    * Flush standard error.
    * Use unified diff.
2018-01-03 13:26:55 -05:00
Umair Khan c2248a81dc Enable test-run-dev. 2018-01-01 09:42:11 -05:00
Umair Khan 9bbce65274 test-run-dev: Redirect stderr to STDOUT.
This redirects the STDERR of run-dev.py command to STDOUT. This is
how you are supposed to redirect according to the docs.
2018-01-01 09:42:11 -05:00
Umair Khan 4c5d361990 test-run-dev: Remove close_and_get_content.
After the new implementation of start_server, this function is not
needed anymore.
2018-01-01 09:42:11 -05:00
Umair Khan 45b004818b test-run-dev: Fix start_server implementation.
Previously, there were following problems with the implmentation:
 * Same file handle was being used to read and write. We used to do
   `seek(0)` and then `read()`. This had a chance to overwrite file
   data. Now we use different file handles to read and write data.
 * We were using text streams. Text streams cannot be used with
   `bufferring=0`. Now we use binary streams without buffering so that
   data is available for reading without any delay.
2018-01-01 09:42:11 -05:00
Umair Khan cd849bc3f1 test-run-dev: Disable Nagios check.
We have moved API to a separate repo. Nagios related code still assumes
that the api folder exists in the Zulip repo.
2018-01-01 09:42:11 -05:00
rht 185fd99816 mypy: Use Python 3 type syntax in several files. 2017-12-30 07:34:51 -05:00
cPhost 470e57db49 node tests: Enforce 100% coverage for stream_data.js. 2017-12-28 09:39:45 -05:00
Steve Howell 9391f7aa98 quick fix: Turn off git lint for Travis builds.
We may be having problems with git lint, so this is a quick
fix to hopefully get builds working again.
2017-12-27 18:20:18 -05:00
Rhea Parekh 6b32519a71 Slack importer: Add Travis tests. 2017-12-27 07:50:53 -05:00
Rhea Parekh 5b37c5562b Slack importer: Add the slack to zulip data conversion files. 2017-12-27 07:50:53 -05:00
Rhea Parekh 6e374d30e1 tools/lib/provision: Clean the code to directly read files in binary mode. 2017-12-26 09:09:31 -05:00
Umair Khan 68513952fb email-worker: Create EmailSendingWorker.
This commit just copies all the code from MissedMessageSendingWorker
class to a new EmailSendingWorker class. All the logic to send an email
through a queue was already there. This commit only makes the logic
generic. It does so by creating a special purpose queue called
'email_senders' to send any type of email. To make
MissedMessageSendingWorker still work we derive it from
EmailSendingWorker. All the tests that were testing
MissedMessageSendingWorker now run against EmailSendingWorker.
2017-12-20 19:36:27 -08:00
Vishnu Ks f743eeb1eb circleci: Set up locale en_US.UTF-8.
See comment for motivation and what's known.

[greg: Added comment, summarizing Vishnu's results from testing.]
2017-12-20 19:18:29 -08:00
Greg Price eb650306eb circleci: Disable gitlint for now.
We get the following error (edited slightly):

  Dec 19 06:13:27 commit_messages| An error occurred while executing
    '/usr/bin/git rev-list --max-count=-1 upstream/master..HEAD':
  fatal: ambiguous argument 'upstream/master..HEAD':
    unknown revision or path not in the working tree.
  Use '--' to separate paths from revisions, like this:
  'git <command> [<revision>...] -- [<file>...]'

We'll need to adjust the remotes to make `upstream` mean what we expect.
2017-12-18 22:21:49 -08:00
Greg Price e5c6c446b5 travis + circleci: Start postgresql during provision.
Otherwise, in CircleCI it isn't running when we need it later.
Travis must special-case starting it automatically.
2017-12-18 22:05:27 -08:00
Greg Price c83c118c6c circleci: Crudely clone a bit of `is_travis` logic.
This isn't really the right way to do this -- commit
  dcd80e665 "travis/setup-backend: Remove the '--travis' flag"
took us in the wrong direction by introducing more magic, deeper in
the stack.

But it's the same way we do it for Travis.  For now, just copy that.

[Thanks to hackerkid for cleaning up my original crude hack.]
2017-12-18 22:04:39 -08:00
Greg Price e8bf7d23bf provision: Remove transitional Python 2->3 logic in setup_venvs.
At this point if we were accidentally using `/srv/zulip-venv` for
anything, we'd have run into it by now.  So just drop the bit of
historical logic that we had to ensure that.
2017-12-18 21:59:51 -08:00
Greg Price 3ba0d5659f travis: Don't create nagios state.
As the comment says, we don't seem to use this.  And the Travis builds
work fine without it.
2017-12-18 21:43:07 -08:00
cPhost 3dc315a0a1 node tests: Add 100% coverage for narrow_state.js. 2017-12-18 09:57:43 -05:00
Tommy Ip 7243a91e47 linter: Re-enable gitlint. 2017-12-18 09:36:47 -05:00
Greg Price 137c0e65bb tools: Revert to Python 2 typing syntax for now.
This reverts commit 66261f1cc.  See parent commit for reason; here,
provision worked but `tools/run-dev.py` would give errors.

We need to figure out a test that reproduces these issues, then make a
version of these changes that keeps that test working, before we
re-merge them.
2017-12-13 10:38:15 -08:00
Greg Price d5fd2a1d72 tools: Fix run-dev.py, which had an `if False:` on a needed import. 2017-12-12 20:50:00 -08:00
rht 66261f1cc3 tools: Use Python 3 syntax for typing in many files. 2017-12-12 17:42:57 -08:00
Symt 3706ec7479 linter: Add good/bad lines for CSS and HTML. 2017-12-11 15:37:22 -06:00
Steve Howell 738c73f08e lint: Prevent `self: Any` annotations.
We should omit these for mypy.  For most class definitions,
mypy doesn't need `Any`, and it provides no real useful info.

For clever monkeypatches, you should provide a more specific
type than `Any`.
2017-12-08 19:00:18 -08:00
cPhost c6985e2eee eslint: add --cache flag for faster linting 2017-12-07 15:14:51 -08:00
picapi_ 96fa399ce0 mypy: Use Python 3 type syntax in tools/lister.py. 2017-12-05 16:32:18 -08:00
Greg Price ff435439af circle-docker: Describe how to build the image.
These commands are super boring standard Docker commands,
so this probably isn't helpful for anyone who is familiar
with building Docker images... but I had to consult docs
to work out the right commands again today, so they'd help me.
2017-12-01 17:16:51 -08:00
Greg Price 535fd9bb5c circle-docker: Silence some spam output. 2017-12-01 17:16:51 -08:00
Greg Price fda349fd33 circle-docker: Fix getting a sudo prompt.
Provision was failing at tools/setup/postgres-init-dev-db
with this in the log:

    Sep 20 02:27:01 + sudo -i -u postgres psql ''
    [sudo] password for circleci:

The issue is that the old version of this line (from Circle upstream)
only lets the `circleci` user sudo to root -- not to other users, or
not directly anyway -- because sudoers syntax is complicated.  Fix it,
after studying `man sudoers`.
2017-12-01 15:21:28 -08:00
Greg Price a1265e09ba circle-docker: Add all the standard APT deps we provision.
This is only an optimization -- if this list is missing anything,
we'll get to it in `provision` in the actual build.  That's important,
because we want an existing image to work fine for testing new
versions of our codebase, including changes that may install more
packages in `provision`.

What this does accomplish is keeping provision's `apt-get install`
fast, by leaving it very little work to do.

The list comes from looking at the APT output during provision in an
actual run without this step, and leaving out two packages which
aren't available at this stage, because we get them from PPAs:
postgresql-9.3-pgroonga and postgresql-9.3-tsearch-extras.
2017-12-01 15:21:28 -08:00
Greg Price ab57756d61 circle-docker: Prune redundant packages from install list for clarity. 2017-12-01 15:21:28 -08:00
Greg Price 642fb8537e circle-docker: Use -scm variant of base image to save a bit of work. 2017-12-01 15:21:28 -08:00
Greg Price 8050ad6c3b circle-docker: Simplify a bit for clarity and efficiency.
Install `jq` with APT -- that's a lot simpler to read than this
explicit download.

And coalesce several commands, following Docker upstream's
recommendation and avoiding unnecessary overhead.
2017-12-01 15:21:28 -08:00
Greg Price 7094f78302 circle: First, near-upstream cut at a Dockerfile.
This is nearly the same as Circle's version, linked in the comment.
I've
* changed the FROM line to get Ubuntu,
* added a couple of distro packages to compensate, and
* revised the comments.
2017-12-01 14:47:55 -08:00
Tim Abbott e9b3ac3f34 travis: Remove now-unnecssary items from apt-mark hold lists.
This seems to have been causing the travis production suite to fail.
It's a direct consequence of removing travis' giant library of apt
sources.list files; now that those are gone, there aren't copies of
all these extra packages available anyway.
2017-11-30 14:56:36 -08:00
Tim Abbott 6d887dc6ee test-backend: Require 100% test coverage on confirmation code. 2017-11-29 22:20:06 -08:00
Greg Price c32b16715d tornado: Use spiffy new `call_later` rather than `add_timeout`.
This method was new in Tornado 4.0.  It saves us from having to get
the time ourselves and do the arithmetic -- which not only makes the
code a bit shorter, but also easier to get right.  Tornado docs (see
http://www.tornadoweb.org/en/stable/ioloop.html) say we should have
been getting the time from `ioloop.time()` rather than hardcoding
`time.time()`, because the loop could e.g. be running on the
`time.monotonic()` clock.
2017-11-29 16:56:29 -08:00
Vishnu Ks 5279ac4601 droplets: Update snapshot id. 2017-11-29 14:24:08 -08:00
Eeshan Garg bae1ecf1c5 tools/setup: Delete static/generated/bots before repopulating.
tools/setup/generate_zulip_bots_static_files now starts off by
deleting static/generated/bots/ (if it hasn't been removed already)
so that outdated static files from older versions of the zulip_bots
package don't supress errors in the main repo that would otherwise
break.

For more info, see #7542

Fixes: #7542.
2017-11-28 17:17:54 -08:00
Greg Price f632692188 lint: Fix use of re.match in judging long lines.
The `re.match` function in the Python stdlib is a trap for the unwary,
with surprising and asymmetrical semantics; we should probably add a
lint rule to ban it entirely.  The docstring says:

> Try to apply the pattern at the start of the string, [...]

In other words, it effectively adds a `^` at the start (or `\A`, where
the distinction matters.)  It's bad enough that this differs from what
grep, sed, perl, less, and every other tool I can think of do when
looking for matches to a regex; on top of that, it treats the
beginning of the string differently from the end, for no obvious
reason.  The function that does what the rest of the world understands
by "match against this regex" is `re.search`.

In this case, it's unlikely that anyone intended for comments with
URLs, or `api_url` references, to miss out on their respective
exceptions to the long-line rule if they happen to start after the
first column.  So fix those rules by just switching to `re.search`
with the same pattern.

I think Markdown URL references may have to start at the beginning of
the line, so I've left a `^` there to preserve -- but now make
explicit -- the `re.match` behavior.
2017-11-28 16:23:10 -08:00
Umair Khan 274bba82b9 two_factor: Add configuration and URLs.
This adds django-two-factor to the project, but held behind
settings.TWO_FACTOR_AUTHENTICATION_ENABLED, so that this has no effect
by default.
2017-11-28 15:21:40 -08:00
Tommy Ip 2d7536bf05 Document the add_mentor.py tool. 2017-11-28 13:26:59 -08:00
Tommy Ip ba1a32ca74 Create script to add and remove mentor's ssh key from DO droplets. 2017-11-28 13:26:59 -08:00
Vishnu Ks 87a412a992 droplets: Fix broken link for requesting droplets. 2017-11-28 08:53:52 -08:00
Vishnu Ks 73ee9a4ad8 droplets: Update info on recreating the droplets. 2017-11-28 08:53:52 -08:00
Vishnu Ks c37193568f droplets: Update baseimage snapshot id.
The new snapshot has python-zulip-env as well
as VIM and mosh installed.
2017-11-28 08:53:52 -08:00
Tommy Ip 19b518c801 refactor: Extract tests for upload mechanics.
This temporarily removes the tests for clear_out_file_list since
fixing that test proved to be difficult.
2017-11-27 21:34:55 -08:00
Tommy Ip ddaff4cd2a refactor: Extract upload mechanics to new JS module.
Tweaked by tabbott to move changes from the next commit that are
required for this to pass tests into this commit.

Note that this exports a few items that were not previously exported.
2017-11-27 21:31:51 -08:00
Eeshan Garg 880fd5497a markdown: Add indented versions of multi-line Markdown macros.
This is part of our efforts to change our integrations/webhooks
docs to follow the same sort of numbered-list format as our /help
docs. In order to indicate that paragraphs separated by newlines
are part of the same numbered-list point, every paragraph must be
indented 4 spaces.
2017-11-25 17:01:11 -08:00
Rhea Parekh 84847b811b requirements: Upgrade mypy to 0.550.
We have to add a few type: ignores due to mypy 0.550 having some bugs
in the new imaplib stubs in typeshed.

Fixes #7387.
2017-11-25 10:06:27 -08:00
Tim Abbott d639e31641 lint: Fix check-templates handling of team.html.
This prevents some unnecessary error messages when running the linter
on a single file.
2017-11-25 09:43:01 -08:00
Tim Abbott 6f53d65e56 docs: Update a few broken links to the Git guide. 2017-11-23 16:24:00 -08:00
Tim Abbott 4d4532a27e copy_and_paste: Add a simple extensible test suite.
This commit is mostly about adding the test suite; some of the results
are things we'd like to change.
2017-11-23 12:59:01 -08:00
Tim Abbott 15501c8b07 lint: Clean up json_rules logic for tab-based whitespace.
This both improves the comment to be more readable, and also uses the
new and improved exclude feature to limit the exclusion to just the
webhook fixtures (where it's needed).

Also fixes a mypy error.
2017-11-23 12:01:20 -08:00
Tim Abbott fb88eaa2d8 lint: Fix logic for exclude to handle directory trees.
The previous exclude rules only allowed excluding a directory (and
things in subdirectories would silently still be linted).  Anyone
using this would expect it to exclude a directory tree, so we make it
do that.
2017-11-23 12:01:20 -08:00
Vishnu Ks def573cde3 remotedev: Add option to recreate droplets. 2017-11-23 11:09:30 -08:00
Tim Abbott 938892db8f lint: Ban tab-based whitespace in our our text fixtures.
We don't want to do this for the webhook fixtures since those come
from third-party code, but it makes sense to do this for hand-edited
code.
2017-11-23 10:47:09 -08:00
Tim Abbott a4e9da22e0 coverage: Update pattern for __str__ for new typing syntax.
The new Python 3 typing syntax means the def lines for __str__ can
look a bit different.
2017-11-22 12:16:58 -08:00
Tommy Ip e44b8809b2 /team: Render contributors list in tabs client side. 2017-11-22 11:33:24 -08:00
Tommy Ip 09d103091d /team: Use list instead of dict for contributors data. 2017-11-22 11:33:24 -08:00
Tommy Ip 771abf4179 /team: Fetch contributors data from all major repos.
Also wait 2 seconds before trying again.
2017-11-22 11:33:24 -08:00
David Rosa Tamsen 11b8b8f483 docs: Add rtd layout template.
The readthedocs theme overrides a few settings in their layout template.
We might want to change some settings back to their default values.

This commit copies the original readthedocs layout file from
https://github.com/rtfd/sphinx_rtd_theme/blob/master/sphinx_rtd_theme/layout.html
to _templates/layout.html, and excludes it from lint and template checks.

Addresses #7417.
2017-11-22 10:59:14 -08:00
Tim Abbott f1850b99cf test-backend: Don't use monkey-patching to set formatter class.
The previous approach threw a mypy error, and was also bad code
anyway.
2017-11-21 21:55:48 -08:00
Tim Abbott 33865a3535 lint: Add exclude rules for URLs in code comments. 2017-11-21 21:53:48 -08:00
rht e55898850a Replace optparse with argparse in remaining tools.
Tweaked by tabbott to fix various bugs with the usage output.
2017-11-21 21:34:38 -08:00
Tim Abbott aa2a4632c5 tests: Lock down 100% test coverage on zerver/forms.py.
With the recent work on OurAuthenticationForm, this now has complete coverage.
2017-11-21 20:14:12 -08:00
Vishnu Ks 766511e519 actions: Mark all messages as read when user unsubscribes from stream.
This fixes a bug where, when a user is unsubscribed from a stream,
they might have unread messages on that stream leak.  While it might
seem to be a minor problem, it can cause significant problems for
computing the `unread_msgs` data structures, since it means we need to
add an extra filter for whether the user is still subscribed, either
in the backend or in the UI.

Fixes #7095.
2017-11-21 20:09:17 -08:00
Vishnu Ks a0275a6257 actions: Send new user messages to signup_notifications_stream. 2017-11-21 17:39:50 -08:00
rht bff736868e Generate custom-icon-webfont on each provision or update-prod-static.
Fixes #7354.
2017-11-20 16:36:49 -08:00
Harshit Bansal 5111aeac41 tools: Rewrite `test-locked-requirements` to be more performant.
This commit modifies `test-locked-requirements` to use some caching
so that we don't need to use the `update-locked-requirements` tool
everytime for checking the validity of locked requirements as it is
slow.

Fixes: #6969.
2017-11-20 15:35:47 -08:00
Harshit Bansal 40fe6239c5 build_emoji: Migrate to python3. 2017-11-20 15:30:58 -08:00
Harshit Bansal df4d10a3df custom_check.py: Fix the comment for ignoring long lines.
Change `#ignorelongline` to `# ignorelongline` as comments must a
have whitespace after `#`.
2017-11-20 15:29:47 -08:00
Harshit Bansal e75f0c1ee4 requirements: Rename requirements files.
This commit renames various source requirements files like `dev.txt`,
`mypy.txt` etc to `dev.in`, `mypy.in` etc and various locked requirements
files like `dev_lock.txt`, `mypy_lock.txt` etc to `dev.txt`, `mypy.txt`
etc. This will help in emphasizing to the user that *.in are actually
input to `update-locked-requirements` tool which should be run after
updating any of these.
2017-11-21 02:38:26 +05:30
Harshit Bansal 95b3f2e745 requirements: Add mypy.txt and docs.txt to locked requirements framework. 2017-11-21 02:38:26 +05:30
Tim Abbott e81b64d511 docs: Move remotedev docs to docs/development/ and fix links.
Thanks to SivagiriVisakan for pointing out the issues here; I ended up
redoing this since the doc really belongs in that directory.
2017-11-20 11:13:59 -08:00
Tim Abbott 25ddba99f6 lint: Ban general use of user_profile.save().
This often can cause minor caching problems.

Obviously, it'd be better if we had access to the AST and thus could
do this rule for UserProfile objects in general.
2017-11-20 10:57:08 -08:00
Balaji2198 c3ac9894a5 docs: Move migration-renumbering.md to subsystems/.
It belongs in that part of the documentation.  We also update the
links in docs/subsystems/schema-migrations.md.
2017-11-20 10:20:31 -08:00
Greg Price 13aa23eb40 provision: Run clean-unused-caches.
This prevents the caches in /srv from growing to fill up the disk --
e.g., on my laptop after 6 months of regular development the venv cache
was 12G and the NPM cache 5G, making them by far the largest disk hogs
on the machine.

It costs about 0.4s, apart from any time spent actually removing
things.  This is a little annoyingly slow to be adding to every
provision, and seems like it could be optimized, but I think already
worth it as is.
2017-11-18 15:04:08 -08:00
derAnfaenger 73915a9631 docs: Move integration docs to /api. 2017-11-17 11:58:54 -08:00
Tim Abbott 57c78da915 test-documentation: Build docs in parallel.
We already do this by default in tools/build-docs, but since we
migrated test-documentation to not run that directly (to disable
collapsing), we need to add the recent parallelism fix here too.

It saves about 5-10s when running this test suite for me, which is
good, but definitely leaves me feeling like there could be more
improvement.
2017-11-17 11:06:56 -08:00
Aditya Bansal dd037df2c4 thumbor: Add dependencies and virtualenv setup script.
In this commit we add new dependencies needed for running thumbor.
Also we add the script for creating the virtual environment ready
for thumbor.
Note: Thumbor will use python2 and thus have different virtualenv
dedicated to it.
Credits to @TigorC and @joshland as well for there work on this.
2017-11-16 22:38:29 -08:00
Aditya Bansal 66060c703c update-locked-requirements: Add py2 option to compile requirements.
In this commit we add a new option which could be used to specify
python version. When 'py2' is specified, future/futures are not
removed from the requirements lock file generated.
2017-11-16 22:34:14 -08:00
Tim Abbott 41a0fd97c4 build-release-tarball: Improve help/error output. 2017-11-16 22:19:58 -08:00
Tim Abbott 780d83efab build-release-tarball: Add support for --help. 2017-11-16 22:19:58 -08:00
Tim Abbott 569cb1d2fb docs: Move readme-symlink symlink to be just called readme.
This name was always a little confusing.
2017-11-16 21:43:25 -08:00
Vishnu Ks 40fca20c51 droplets: Run git clean -f after creating droplet. 2017-11-16 19:47:42 -08:00
Vishnu Ks 75601f660d droplets: Always rebase instead of merge during git pull. 2017-11-16 19:47:42 -08:00
Greg Price 0cc8d243e4 build-release-tarball: Fold away the confusing variable TMP_CHECKOUT.
Even where this is actually used for a temporary checkout, it obscures
the relationship between this and $TMPDIR -- and some of our logic
depends on that.  In other places, it isn't actually even a checkout.
In all cases, the expanded version is clearer.
2017-11-16 15:54:34 -08:00
Greg Price 3d1211edcf build-release-tarball: Cut a recently-obsoleted comment.
We no longer use .gitattributes for this purpose (since bfaf0fb3b),
so this comment isn't true anymore.
2017-11-16 15:54:34 -08:00
Greg Price 18f9e18324 build-release-tarball: Drop likely-bitrotted macOS support.
This script, and tools/update-prod-static which it relies on,
have kept getting more complex since this conditional was added
in 2013, and the places we rely on GNU features have probably
multiplied beyond `mktemp -d`.  It's unlikely this works on
macOS with BSD tools now, and it'd be hard to maintain that way
if it did; drop the pretense.
2017-11-16 15:54:34 -08:00
Greg Price 5a2f3fe9c2 build-release-tarball: Reorder a bit for clarity.
This puts our fiddling with the update-prod-static logfiles
immediately after we run update-prod-static.
2017-11-16 15:54:34 -08:00
Greg Price 93fbffebd7 build-release-tarball: Cut a redundant cleanup step.
There's no need to remove this file here -- the whole tree will be
removed a few commands later, and the `tar` command we do first, to
supplement our tarball with various generated files, is quite
selective and wouldn't look at this file anyway.
2017-11-16 15:54:34 -08:00
Greg Price 09a38ec5dc build-release-tarball: Cut a long-stranded comment.
This comment only made sense for about one week, in 2013;
the setting it points at was removed in e6ca5d265.
2017-11-16 15:54:34 -08:00
Greg Price 8d405da96c build-release-tarball: Use set +x consistently. 2017-11-16 15:54:34 -08:00
Greg Price 92e322a9d1 build-release-tarball: Add --authors-not-required.
This makes it not real annoying to run this in a development loop.
2017-11-16 15:54:34 -08:00
Tim Abbott e1f8d24166 lint: Fix tools tests.
These were broken when Travis wasn't running properly.
2017-11-16 13:59:40 -08:00
Tim Abbott 2563f93d46 test-help-documentation: Pass use_db=True.
With the new portico work we've done, the help documentation does
sorta depend on the database if you're logged in.  So it's best to
just require it for these tests.
2017-11-16 13:44:53 -08:00
Tim Abbott cb4acf2aa4 docs: Lint rst files for trailing whitespace. 2017-11-16 13:20:50 -08:00
Tim Abbott d9da8f859d lint: Check for absolute/relative link style in developer docs.
This should help avoid a pretty wide range of link bugs we've had in
the past.
2017-11-16 12:44:16 -08:00
Tim Abbott bf77ad3a6e docs: Clean a few files of too-long lines.
We also document why the rest of them are still on the list.
2017-11-16 11:49:00 -08:00
Tim Abbott 7c24e04223 lint: Fix long lines in README.md. 2017-11-16 11:38:13 -08:00
Tim Abbott 638eb7a8e4 docs: Update links to ReadTheDocs to always use https.
This is better security practice.

We also add a lint rule to enforce this for the future.
2017-11-16 10:59:24 -08:00
Tim Abbott 054952a44a docs: Update links from codebase to point to ReadTheDocs. 2017-11-16 10:53:49 -08:00
David Rosa Tamsen 7072fa5b37 docs: Reorganize developer docs to improve navigation.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.

This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.

Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).

Fixes #5265.
2017-11-16 09:45:08 -08:00
Tim Abbott 774380dcb8 capitalization: Allow enabled/disabled to be lower-case. 2017-11-15 17:39:09 -08:00
Tommy Ip 0a445031a9 /team: Remove responsive columns hack.
Since we are restyling the contributors table to the new fancy CSS
grid layout we no longer need this hack.
2017-11-15 17:08:08 -08:00
rht e54b9730fc Reduce python files line length limit to 110.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-15 10:58:03 -08:00
rht bafba95eb7 Text-wrap the remaining long lines exceeding 110. 2017-11-15 10:58:03 -08:00
Eeshan Garg e85d0747d4 tools/test-help-documentation: Verify api/ docs. 2017-11-15 10:12:59 -08:00
derAnfaenger 5ef09bfcd4 docs: Move bot docs to /api.
With a few fixes for rendering bugs.  The rest will be in a follow-up
issue.
2017-11-15 09:58:29 -08:00
Vishnu Ks 36f29764cb emails: Add option to forward mails send in dev env to external email.
Fixes #7085.
2017-11-14 15:27:29 -08:00
Tim Abbott ac763d6eed tools: Remove obsolete build-deb tool.
This hasn't been used in ages, and isn't useful with the current
codebase.
2017-11-14 15:22:26 -08:00
rht 72ce1c6501 Use a more concise `mkdir -p` when the situation permits. 2017-11-14 09:47:09 -08:00
Steve Howell ba51078418 Simplify CSS linter and clean up CSS.
The CSS linter was pretty hard to reason about.  It was
pretty flexible about certain things, but then it would
prevent seemingly innocuous code from getting checked in.

This commit overhauls the pretty-printer to be more composable,
where every object in the AST knows how to render itself.  It
also cleans up a little bit of the pre_fluff/post_fluff logic
in the parser itself, so comments are more likely to be "attached"
to the AST node that make sense.

The linter is actually a bit more finicky about newlines, but
this is mostly a good thing, as most of the variations before
this commit were pretty arbitrary.
2017-11-13 12:43:43 -08:00
Vishnu Ks 50eb3be0c2 droplets: Remove ip_address default value from create_dns_record.
I forgot to remove the default value 172.31.1.4 from
ip_address which was used only for testing purpose.
2017-11-11 17:05:14 -08:00
rht f397e15128 tools: Text-wrap long lines exceeding 110. 2017-11-10 16:24:09 -08:00
rht 7a96b8c7ad tools/lib: Text-wrap long lines exceeding 110. 2017-11-10 16:24:09 -08:00
Vishnu Ks 07e6a4fb2d droplets: Add wildcard A records to support realm subdomains. 2017-11-10 16:13:53 -08:00
rht 5ff690d486 zulip-export: Convert to using argparse from optparse. 2017-11-10 15:54:02 -08:00
rht f689813f38 test-js-with-node: Port from optparse to argparse. 2017-11-10 15:52:43 -08:00
Cynthia Lin 0b800b0a7d icons: Create framework for custom icons and add new bot icon. 2017-11-10 11:18:42 -08:00
neiljp (Neil Pilgrim) fec59b29cd Tools/mypy: Enforce typed generics in tools/run-mypy.
This adds the "--disallow-any=generics" option to run-mypy, which no
longer permits:
- inheriting from "list"; use "List[sometype]" (or a TypeVar)
- generic types with no following square brackets specifying the type
  (even if initially 'Any')

Any (and '...' for Callable) is a lot easier to search for than an
absence of square brackets, and should improve overall typing quality.
2017-11-08 12:40:40 -08:00
derAnfaenger c9d438d149 linter: Add example lines for several python rules. 2017-11-08 11:18:37 -08:00
derAnfaenger dbc1f83d3a linter: Make string formatting custom rule more complete. 2017-11-08 11:16:18 -08:00
derAnfaenger 88794ea361 linter: Fix tests ignoring rules with `include_only`. 2017-11-08 11:16:18 -08:00
rht 9c7d5812ce refactor: Remove six.moves.urllib.parse import. 2017-11-07 10:51:44 -08:00
rht ccf2792c1c refactor: Remove six.moves.configparser import. 2017-11-07 10:51:44 -08:00
rht 6b12fe24b6 refactor: Remove six.moves.input import. 2017-11-07 10:51:44 -08:00
rht 6cce0e346e refactor: Remove six.moves.filter import. 2017-11-07 10:51:44 -08:00
rht 80a8d4f9f3 refactor: Remove six.moves.map import. 2017-11-07 10:46:42 -08:00
rht 549a26860f refactor: Remove six.moves.range import. 2017-11-07 10:46:42 -08:00
rht ec5120e807 refactor: Remove six.moves.zip import. 2017-11-07 10:46:42 -08:00
Vishnu Ks c5ff020246 droplets: Garbage collect A records before creating droplet. 2017-11-06 12:20:39 -08:00
Rishi Gupta dec4b9ed93 remote dev: Add code and instructions for creating digital ocean droplets.
Mostly copied from the zulip/zulip-gci repository, but with some changes to
wordings and code cleanup for linters.
2017-11-06 12:20:39 -08:00
rht e3eebf3be0 Remove inheritance from object. 2017-11-06 08:53:48 -08:00
neiljp (Neil Pilgrim) 737408f741 mypy: Add explicit flexible type parameters for Callable in run-dev.py. 2017-11-04 19:47:44 -07:00
neiljp (Neil Pilgrim) 9f90a3174b mypy: Add explicit Any as Dict parameters in check_line_coverage in test-js-with-node. 2017-11-04 19:47:44 -07:00
neiljp (Neil Pilgrim) abc0d5e79b mypy: Add type parameter for Set in check_help_documentation.py. 2017-11-04 19:47:44 -07:00
neiljp (Neil Pilgrim) a5d1fdf0d6 mypy: Add type parameter for IO in test-queue-worker-reload. 2017-11-04 19:47:44 -07:00
neiljp (Neil Pilgrim) dac67ebe67 mypy: Remove un-necessary comment re mypy issue #3145.
RuleList is required at global scope now, for later functions.
2017-11-04 19:22:06 -07:00
neiljp (Neil Pilgrim) 05ef052ef5 mypy: Improve typing in custom_check_file() in custom_check.py. 2017-11-04 16:18:27 -07:00
rht dcc831f767 refactor: Replace all __unicode__ method with __str__.
Close #6627.
2017-11-02 11:01:47 -07:00
Tim Abbott d27002ba21 landing-page: Replace /about with /team and /history in links. 2017-10-31 12:44:05 -07:00
neiljp (Neil Pilgrim) 966d5f3760 mypy: Specify Callable parameter/return type for run decorator in tools/diagnose. 2017-10-31 00:03:35 -07:00
Tim Abbott f01f596138 help: Merge api.js and help.js bundles to fix code blocks.
In addition to decreasing the excessive number of bundles we had, this
will set us up to fix rendering of code blocks when clicking the
sidebar links in the /api-new site.
2017-10-30 22:53:17 -07:00
Brock Whittaker e86e259cb1 /api/: Add Markdown-based /api-new documentation.
This commit allows for the /api-new/ page to rendered similarly to our
/help pages.  It's based on the old content for /api, but we're not
replacing the old content yet, to give a bit of time to restructure
things reasonably.

Tweaked by eeshangarg and tabbott.
2017-10-30 22:32:43 -07:00
Greg Price eb55a3a1ba template context: Give better names to the URLs for the API.
The "subdomain" label is redundant, to the extent it's even
accurate -- this is really just the URL we want to display,
which may or may not involve a subdomain.  Similarly "external".

The former `external_api_path_subdomain` was never a path -- it's a
host, followed by a path, which together form a scheme-relative URL.
I'm not quite convinced that value is actually the right thing in
2 of the 3 places we use it, but fixing that can start by giving an
accurate name to the thing we have.
2017-10-30 18:29:29 -07:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
Brock Whittaker 8c9579eb86 webpack.assets.json: Add a custom JS bundle for /api.
Note that we also need the JS bundle for /help because that
includes the JS code for our sidebar dropdowns.
2017-10-29 18:18:32 -07:00
Harshit Bansal 1067cfd92a node_modules: Add npm packages for the supported emojisets.
This commit adds emoji datasource packages for the rest of the emojiset
packages which we will use for serving images in notifications.
2017-10-28 10:38:32 -07:00
Tim Abbott 0d1194811f mypy: Remove ignores for a few typeshed bugs fixed upstream. 2017-10-27 17:09:00 -07:00
Tim Abbott 8d00111a27 vagrant: Skip the shell virtualenv warning on provision. 2017-10-27 11:44:47 -07:00
rht e296841447 tools/documentation_crawler: Use python 3 syntax for typing. 2017-10-26 21:58:22 -07:00
Tim Abbott 8e2cdedf9a lint: Fix lines in Python codebase longer than 120 characters. 2017-10-26 17:47:30 -07:00
Tim Abbott b936e8c24b lint: Fix lines in Python codebase longer than 125 characters. 2017-10-26 17:36:54 -07:00
Tim Abbott be619fe881 lint: Wrap many very long lines in the Python codebase.
This decreases the maximum line length in our Python codebase to 130.
2017-10-26 17:31:58 -07:00
Greg Price 4a6e867046 lint: Disable a buggy `pep8` rule on line breaks and operators.
I'd much rather see something like

    if (thing_is_permissible(user, thing)
            or (user_possesses_hammer(user)
                and glass_break_requested(thing))):

than

    if (thing_is_permissible(user, thing) or
            (user_possesses_hammer(user) and
                glass_break_requested(thing))):

because the former makes the overall logic much easier to scan.
Similarly for a formula full of arithmetic rather than Boolean
operators.  And the actual PEP 8 agrees (though until 2016 it
unfortunately had the opposite advice.)

The upstream linter still applies the backward rule, so disable that.
2017-10-26 10:29:17 -07:00
Brock Whittaker a9a9333b2a portico: Show whether the user is logged in.
This creates a dropdown in place of the normal register/login links
you get when logged out, with an option to go to the app or log out if
that appears you click on the avatar.

A bit more work is needed to make this look really good, but it's a
great start.
2017-10-25 14:59:24 -07:00
Aditya Bansal 0ddad98c98 features.html: Cleanup to use 4 space and consistant indentation. 2017-10-25 10:58:57 -07:00
Aditya Bansal 3fb0f7c092 home.html: Cleanup to use 4 space indentation.
Tweaked by tabbott to also make the translation tags make more sense.
2017-10-25 10:58:49 -07:00
Tim Abbott 618ace1f5c build-release-tarball: Fix missing CSS-inlined email templates.
These apparently we're being included.
2017-10-20 14:08:25 -07:00
Tim Abbott c9e29b1c54 check-templates: Document status of remaining bad_files items. 2017-10-19 16:56:46 -07:00
Tim Abbott 2fb46df005 check-templates: Remove exclude rules for removed HTML files.
These webhook integrations were converted to using markdown some time
ago.
2017-10-19 16:56:46 -07:00
Tim Abbott fc4b87ed04 templates: Clean some whitespace issues in markdown_help.
Unfortunately, the rest cannot be easily fixed.
2017-10-19 16:56:46 -07:00
Tim Abbott e80fc93e87 templates: Fix search_operators indentation. 2017-10-19 16:56:46 -07:00
Tim Abbott 59344d77df templates: Clean various items off HTML linter bad list.
Most of these already pass the linter; for the ones that don't, the
fixes are small and don't make the HTML any worse.
2017-10-19 16:56:00 -07:00
Tim Abbott 540cae19a8 puppet: Remove obsolete sparkle configuration.
Sparkle was the auto-update system used by the legacy desktop app.  We
haven't been capable of using it for auto-update in years, so there's
no reason to keep around the configuration.

The new Electron app uses a different system anyway.
2017-10-19 16:35:55 -07:00
Tim Abbott 4ddb812354 check_subdomain: Remove case for user_subdomain being None.
With REALMS_HAVE_SUBDOMAINS always on, user_subdomain is never None,
so this was just dead code.
2017-10-18 22:35:07 -07:00
Tim Abbott 1ab2ca5986 subdomains: Extract zerver.lib.subdomains library.
These never really belonged with the rest of zerver.lib.utils.py, and
having a separate library makes it easier to enforce full test
coverage.
2017-10-18 22:27:48 -07:00
rht c8090aafa4 lint: Remove several unused imports. 2017-10-18 10:56:11 -07:00
Tim Abbott fe3a8084d8 mypy: Fix annotations for linter. 2017-10-18 10:50:25 -07:00
Tim Abbott 2bc14d256f lint: Ban two spaces after comma in JS code.
We exclude the frontend tests, mostly because the lint rule isn't that
precise, and the test code has some sample user input that's a bit
funny.
2017-10-18 10:22:18 -07:00
Tim Abbott 8afe9835f6 lint: Ban two spaces after comma in prose. 2017-10-18 10:12:41 -07:00
Tim Abbott b634d9f38b lint: Ban two spaces after comma in CSS. 2017-10-18 10:11:33 -07:00
derAnfaenger 194e102077 linter: Add rule against multiple whitespaces after comma. 2017-10-18 10:04:23 -07:00
rht 691598a88b py3: Remove "from six.moves import range".
This is no longer required, since in Python 3, this is what the range
built-in does.
2017-10-17 23:28:14 -07:00
rht a603a4f9f5 Remove `from __future__ import absolute_import`.
Except in:
- docs/writing-bots-guide.md, because bots are supposed to be Python 2
  compatible
- puppet/zulip_ops/files/zulip-ec2-configure-interfaces, because this
  script is still on python2.7
- tools/lint
- tools/linter_lib
- tools/lister.py

For the latter two, because they might be yanked away to a separate repo
for general use with other FLOSS projects.
2017-10-17 22:59:42 -07:00