Commit Graph

3703 Commits

Author SHA1 Message Date
Mayank Madan 264c2e9803 provision: Add provision support for Ubuntu 18.10.
Note that this is only support for Ubuntu 18.10 in the development
environment.
2019-03-09 16:01:46 -08:00
vinitS101 4523c8d7ee typo: Change "generage" to "generate". 2019-03-09 15:49:38 -08:00
Tim Abbott 0ab8cd9ca3 provision: Refactor handling of dependencies for building pgroonga.
This lets us leave build-pgroonga platform-agnostic.
2019-03-08 15:12:56 -08:00
Tim Abbott 5c20ee998c build-tsearch-extras: Add support for Debian/Ubuntu. 2019-03-08 15:12:56 -08:00
Tim Abbott 32d26c8aad provision: Clean up building search extensions from source.
This generalizes the provision logic for deciding whether to build our
tsearch_extras and pgroonga search extensions from source to support
Ubuntu cosmic as well (and evenutally, other future platforms).
2019-03-08 15:12:56 -08:00
Tim Abbott 8469c7774f scripts: Remove create-zulip-admin.
This was used only by ancient versions of the Docker project.
2019-03-04 15:33:51 -08:00
Tim Abbott 500b5cdaf1 tools: Fix missing comma in collectstatic configuration.
It appears that this code did the right thing despite being written
wrong, probably due to whatever `manage.py collectstatic` does in its
argument parsing.  But in any case, we should make the code read how
it's intended.
2019-03-04 08:31:12 -08:00
Rohitt Vashishtha 3d427d02cc scripts/zulip_tools: Use run_as_root instead of subprocess.check_call. 2019-03-01 11:21:16 -08:00
Rohitt Vashishtha ac48925977 scripts: Use run_as_root instead of run([sudo, ...]). 2019-03-01 11:21:16 -08:00
Wyatt Hoodes 5231b27dea popovers: Add a clear status option in user popover.
Accomplished by adding a function to clear the status message with
an empty string. The html is then updated to reflect changes without a
refresh.

Currently, it's a small hassle to clear a status message.  This option
makes things a bit easier.

Fixes #11630.
2019-02-26 14:40:17 -08:00
Anders Kaseorg 9faa009f66 lint: Prohibit unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:50 -08:00
Anders Kaseorg f1ec67c614 python: Mark intentionally unused imports with noop statements.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-22 16:54:47 -08:00
Wyatt Hoodes 2580965284 test-backend: Rename '--process' option to '--parallel'.
This change is being made to simply match up our naming conventions
with django's test framework.
2019-02-20 13:25:49 -08:00
Raymond Akornor 4dc7f5354d tests: Optimize test-backend performance.
This optimizes test-backend by skipping webhook
tests when run in default mode.

Tweaked by tabbott to extend the documentation and update the CI
commands.
2019-02-19 14:50:41 -08:00
Tim Abbott fe4a0d0fcd capitalization: Fix night logo lint errors. 2019-02-18 15:53:55 -08:00
Steve Howell e67cf30dfd private messages: Add user circles to top left.
This is mostly adding markup, calling some convenient
functions in buddy_data.js, and adjusting CSS.

To make the circles update dynamically, I mostly
orchestrate this though activity.js for now.  It's
possible we'll want to adjust that eventually to
happen through something like a `presence_events`
dispatcher, but that's essentially what
a good part of `activity.js` does now.
2019-02-18 14:22:37 -08:00
Anders Kaseorg dec8c7dba4 provision: Run generate_secrets earlier.
Commit 7d12e2019d (#10994) broke fresh
provisions by importing zproject.settings before we were done
modifying settings.  Fix it by moving the generate_secrets invocation
to the earliest reasonable place.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-08 10:13:18 -08:00
Tim Abbott a2373a63cd capitalization: Add GCM to list of proper nouns.
This fixes failures in master from the recent GCM merge.
2019-02-08 10:11:55 -08:00
Anders Kaseorg 8fdb62a6bc tools: Create DEPLOY_ROOT/zulip-py3-venv symlink in development too.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-07 17:12:33 -08:00
Vishnu Ks b62bd83083 invites: Add generate multiuse invite button. 2019-02-07 15:41:00 -08:00
Rishi Gupta 36472413e4 hotspots: Add hotspot for gear menu. 2019-02-05 12:19:21 -08:00
Vishnu Ks 4b1229c072 billing: Add node tests for upgrade.js. 2019-02-04 22:37:05 -08:00
Anders Kaseorg ee8ff4df66 tools: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:10:31 -08:00
Hemanth V. Alluri 73d26c8b28 streams: Render and store the stream description from the backend.
This commit does the following three things:
    1. Update stream model to accomodate rendered description.
    2. Render and save the stream rendered description on update.
    3. Render and save stream descriptions on creation.

Further, the stream's rendered description is also sent whenever the
stream's description is being sent.

This is preparatory work for eliminating the use of the
non-authoritative marked.js markdown parser for stream descriptions.
2019-02-01 22:24:18 -08:00
Anders Kaseorg 893e1475bb setup_virtualenv: Do not activate the virtualenv on creation.
Instead, manually activate it in the one place where this
functionality was used (tools/lib/provision.py).  This way we avoid
trying to activate the Python 2 thumbor virtualenv from Python 3.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-31 17:55:04 -08:00
Greg Price 6613f4c22c test-install/install: Print usage when run without arguments.
The `-z "$INSTALLER"` was intended to do this -- but we don't get that
far, because the `shift` fails.
2019-01-31 16:15:51 -08:00
Steve Howell c665f809f2 run-dev: Make address-in-use errors more obvious.
This seems to be a common enough pitfall to justify
a bit of extra handling.  Example output:

    $ ./tools/run-dev.py
    Clearing memcached ...
    Flushing memcached...
    OK
    Starting Zulip services on ports: web proxy: ...
    Note: only port 9991 is exposed to the host in a Vagrant environment.

    ERROR: You probably have another server running!!!

    Traceback (most recent call last):
      File "./tools/run-dev.py", line 421, in <module>
        app.listen(proxy_port, address=options.interface)
      File "/srv/zulip-py3-venv/lib/python3.5/...
        server.listen(port, address)
      File "/srv/zulip-py3-venv/lib/python3.5/...
        sockets = bind_sockets(port, address=address)
      File "/srv/zulip-py3-venv/lib/python3.5/...
        sock.bind(sockaddr)
    OSError: [Errno 98] Address already in use
    Terminated
2019-01-29 16:03:47 -08:00
Steve Howell 1228e541d7 dispatch test: Restore 100% line coverage (user_status).
We now have 100% coverage on server_events_dispatch.js.

The last piece were was covering "status_text" updates
in the "user_status" event.
2019-01-26 10:48:57 -08:00
Steve Howell bacf896228 poll widget: Clean up code and add edit controls.
NOTE: If you revert this commit, you want to revert
the immediately prior commit as well.  The history
is that Ishan made some improvements to the widget,
but there were some minor bugs.  I decided not
to squash the commits together so that the git
history is clear who did what.  (In particular, I
want questions about the JS code to come to me if
somebody does `git blame`.)

Anyway...

This is a fairly significant rewrite of the polling
widget, where I clean up the overall structure of
the code (including things from before the prior
fix) and try to polish the prior commit a bit as
well.

There are a few new features:

    * We tell "other" users to wait for the poll
      to start (if there's no question yet).
    * We tip the author to say "/poll foo" (as
      needed).
    * We add edit controls for the question.
    * We don't allow new choices until there's
      a question.
2019-01-22 10:27:39 -08:00
Harshit Bansal 2944075f70 lint: Respect `--force` flag while running mypy. 2019-01-17 10:34:43 -08:00
Anders Kaseorg 25d6107a4f docs: Avoid hardcoded /tmp paths in miscellaneous documentation.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:16:12 -08:00
Anders Kaseorg 79d888223f profiling: Use mkstemp for profile.data filename.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:08:48 -08:00
Anders Kaseorg f3c9b87c30 test-locked-requirements: Avoid hardcoded paths in /var/tmp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:05:51 -08:00
Anders Kaseorg b4e1403cf9 test-install: Avoid hardcoded paths in /tmp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:05:51 -08:00
Anders Kaseorg c17ddceeae test-emoji-name-scripts: Avoid hardcoded paths in /var/tmp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:05:50 -08:00
Anders Kaseorg df7b63cd5e tools: Remove check-css.
As of commit 0042694e24 (#10017), this
is unused.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:05:50 -08:00
Anders Kaseorg f099fd9e28 pretty_print: Replace hardcoded path in /var/tmp with a pipe.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:00:48 -08:00
Anders Kaseorg 6678a95b20 test-queue-worker-reload: Use a pipe rather than polling a log file.
This avoids unnecessary polling delays, quadratic string operations,
and hardcoded paths in /tmp.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:00:48 -08:00
Anders Kaseorg 19429c3ad7 test-run-dev: Use a pipe rather than polling a log file.
This avoids unnecessary polling delays, quadratic string operations,
and hardcoded paths in /tmp.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:00:48 -08:00
Tim Abbott e29533df0f emails: Fix duplicate <html> blocks in missed-message emails.
Apparently, the missed-message templates have a slightly different
structure from our other email templates, which triggered a latent,
subtle bug in inline-email-css's effort to remove duplicate <html>
blocks from emails that had been generated by premailer.  Fix this
bug, and add appropriate assertions to prevent similar issues in the
future.

Fixes #11249.
2019-01-11 14:23:30 -08:00
Anders Kaseorg 2ad1a27b28 create-production-venv: Remove Python 2 support.
We still create a Python 2 virtualenv for thumbor but that’s
separate (/srv/zulip-thumbor-venv from
scripts/lib/create-thumbor-venv).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-09 17:32:05 -08:00
Vishnu Ks c263674507 billing: Add node test for billing.js. 2019-01-08 10:47:35 -08:00
rht 15763f8545 provision: Include DISTRIB_FAMILY in parse_lsb_release output. 2019-01-07 18:52:09 -08:00
Vishnu Ks 13de959121 tools: Add script to automate python dependency upgrade. 2019-01-07 10:25:35 -08:00
Marco Burstein 9ddadd39f4 compose: Add support for using Zoom as the video chat provider.
This adds Zoom call properties to the `Realm` model, creates endpoints
for creating calls, adds a frontend and tests.

Fixes #10979.
2019-01-07 10:00:02 -08:00
Mateusz Mandera 36714342a3 lint: Allow usage of the word subject in test_email_mirror.py. 2019-01-05 15:59:19 -08:00
Tim Abbott 99721755e8 node: Temporarily remove composebox_typeahead from fully covered. 2019-01-05 15:31:30 -08:00
Tim Abbott 453b65bde3 provision: Clean up comments and unused variables.
This has some changes inspired by #8514.
2019-01-04 15:16:19 -08:00
rht 7e1520b01f provision: Ensure `python3` exists in PATH for CentOS. 2019-01-04 14:11:03 -08:00
Vishnu Ks d94b0da3c8 billing: Separate billing.js into multiple modules. 2019-01-04 23:52:00 +05:30
Vishnu Ks 81e6a022e6 emails: Inline CSS in missed_message.
As part of this change, we port into the .messages class the work in
4e8e7348da to change overflow-y to auto,
not scroll (skipping that would result in a regression).
2019-01-03 17:55:29 -08:00
Vishnu Ks 4e1c0589c2 emails: Refactor if condition from style tag in missed_message_email. 2019-01-03 17:55:29 -08:00
Tim Abbott 4df618f584 portico: Order jquery.validate before signup.js.
This is important, because the signup.js module uses jquery.validate.
2019-01-03 15:22:57 -08:00
Steve Howell d8d703af45 frontend: Add basic user_status module.
So far this processes page_params, but it's otherwise
an unused internal API.
2019-01-02 09:16:31 -08:00
Steve Howell 4d5ce0b285 lint: Exclude "subject" from JS files (except util.js).
We also exempt the frontend tests for now.
2019-01-01 20:52:07 -08:00
Steve Howell b4bc6b4445 emails: Fix subject -> email_subject in send_email.py. 2019-01-01 20:49:00 -08:00
Steve Howell 4da28efc90 emails: Fix subject -> email_subject in error_notify.py. 2019-01-01 20:48:59 -08:00
Steve Howell 16328732d4 refactor: Remove "subject" from stream_topic.py. 2019-01-01 20:48:56 -08:00
rht d3139266c8 setup_venv: Uninstall typing on Python >3.4.
Otherwise this causes an error
```
AttributeError: type object 'Callable' has no attribute '_abc_registry'
```
on 3.7. While the error is specific to 3.7, it is safer to uninstall
typing for all the versions that don't require a pip-provided typing
library.
2019-01-01 19:34:48 -08:00
rht c190959b58 provision: Move CentOS & Fedora venv deps construction to setup_venv.py.
They will be reused in production setup.
2019-01-01 19:34:46 -08:00
rht 17ffbdabe2 provision: Use virtualenv instead of python-virtualenv on Fedora.
The package name changed at some point.
2019-01-01 19:34:24 -08:00
rht 75134ef614 provision: Use generic python3 on Fedora dependencies.
It might be cleaner to add new variables in setup_venv.py, but we can
do that later.
2019-01-01 19:34:11 -08:00
Tim Abbott 5a8ad84381 test-documentation: Exclude the 500ing Citizen Code of Conduct.
We might want to eventually remove this, but I'm holding out hope that
it'll be fixed.
2018-12-29 15:30:19 -08:00
Vishnu Ks 6f5b1903c4 docs: Update code-reviewing docs to reflect CI changes. 2018-12-29 15:28:26 -08:00
Steve Howell 3fc8597119 popovers: Avoid passing message to actions popover template.
We instead get the specific fields from message
that we use.  This is particularly helpful
for subject -> topic migration; we no longer
have to account for "subject" fields in
client-side templates.
2018-12-29 14:19:18 -08:00
Sumanth V Rao ccd0badeed test-documentation: Simplify --skip-external-links code path.
The crawler used to be called directly for checking external links.
Now the scrapy command calls the crawl_with_status wrapper.
Crawl_with_status has been modified to pass the external parameter in
the previous commit, so we can now use this simpler approach.
2018-12-29 09:09:18 -08:00
Sumanth V Rao 776b981dde test-help-documentation: Skips checking external links.
This option causes test-help-documentation to verify internal links only.
This prevents documentation from flaking in CI due to links from
external websites having failed in the documentation.

Based on conversation:
https://chat.zulip.org/#narrow/stream/43-automated-testing/subject/quora.20404/near/674997
2018-12-29 20:15:30 +05:30
Rishi Gupta e7220fd71f billing: Do subscription management in-house instead of with Stripe Billing.
This is a major rewrite of the billing system. It moves subscription
information off of stripe Subscriptions and into a local CustomerPlan
table.

To keep this manageable, it leaves several things unimplemented
(downgrading, etc), and a variety of other TODOs in the code. There are also
some known regressions, e.g. error-handling on /upgrade is broken.
2018-12-22 13:39:30 -08:00
Rishi Gupta f8720fddd6 emails: Update text of notify_new_login. 2018-12-20 16:26:19 -08:00
Vishnu Ks f9e4d5d28e emails: Tag password_reset for translation. 2018-12-19 09:21:09 -08:00
Vishnu Ks 0960e654ab emails: Tag notify_new_login email for translation. 2018-12-19 09:21:09 -08:00
rht acbb174100 provision: Add RHEL 7 support. 2018-12-18 17:13:56 -08:00
rht 84c968ab36 provision: Include scripts/lib/setup-* and scripts/lib/build-* for sha1 sum check. 2018-12-18 16:56:27 -08:00
Tim Abbott dd946fff1f lint: Improve ui_report.error linter rule.
We now allow passing "" as the first argument.
2018-12-18 12:33:15 -08:00
rht a093d45064 provision: Check for pg_hba.conf existence for doing postgres initdb.
Apparently, an empty pg_datadir is initialized after postgresql-10 is
installed.
2018-12-18 12:07:45 -08:00
rht 3060ee9397 provision: Fix typo when checking that vendor is Fedora. 2018-12-18 12:07:03 -08:00
Anders Kaseorg 392175d6e8 Use #!/usr/bin/env for bash shebangs.
/bin/sh and /usr/bin/env are the only two binaries that NixOS provides
at a fixed path (outside a buildFHSUserEnv sandbox).

This discussion was split from #11004.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-17 17:21:08 -08:00
rht 5e12eb8d34 provision: Fix typo in list of services on RedHat. 2018-12-17 16:30:48 -08:00
rht 42cbf2071b provision: Skip RedHat postgresql initdb if already initialized.
We only need to initialize a postgres database once in the lifetime of
a system.
2018-12-17 16:29:50 -08:00
rht 295cbe7b9c provision: Add build-pgroonga and use it for Fedora. 2018-12-17 16:23:57 -08:00
rht 80ba105be5 provision: Abstract out SYSTEM_DEPENDENCIES. 2018-12-17 16:23:46 -08:00
rht b732fe819e provision: Add Fedora support. 2018-12-17 16:23:44 -08:00
Tim Abbott d469c14d11 docs: Advertise arbitrary org GitHub patterns for linkifiers.
Now that we support this, we should make it really obvious that we do.

Also, link to the Help Center article.
2018-12-17 12:58:52 -08:00
rht 2dc1dc8ec2 provision: Replace overwrite_symlink with `sudo ln -sf`.
overwrite_symlink doesn't work for a non-root user when accessing root
content.
2018-12-17 11:50:15 -08:00
Tim Abbott 2558f101af docs: Add documentation for `if False` mypy pattern in scripts.
This should help make it clear what's going on with these scripts.
2018-12-17 11:12:53 -08:00
Tim Abbott 209df75ffa lint: Enforce use of Python 2 style typing in zulint.
This is intended to be extracted as an external project that supports
Python 2+3.
2018-12-17 11:00:08 -08:00
Tim Abbott 35901cc955 lint: Fix a tooling test failure in last commit.
This is what I get for copy-pasting lint rule blocks.
2018-12-17 10:53:08 -08:00
Tim Abbott bce90a3340 lint: Add lint rule for scripts importing typing improperly.
This is a common bug that users might be tempated to introduce.

And also fix two instances of this bug that were present in our
codebase, including an important one in our upgrade code path.
2018-12-17 10:46:37 -08:00
Tim Abbott 6b69cc0b39 lint: Run mypy as part of main linter.
To support this, we add a pass_targets option to the main linter
library, because with current mypy, it's generally counterproductive
to pass the list of files in (can produce spurious errors; isn't
faster).
2018-12-16 21:52:53 -08:00
Tim Abbott 3003517430 docs: Move mypy documentation from contributing to testing.
The testing section is more appropriate, since it's fundamentally part
of our CI system.

While we're at it, fix the fact that we were linking to GitHub, not
ReadTheDocs, in the run-mypy output.
2018-12-16 21:52:53 -08:00
Tim Abbott 8637bdeb30 mypy: Provide a clear message when starting daemon.
Previously, when the mypy daemon was starting up, it was very unclear
from the linter context what was happening.
2018-12-16 21:52:53 -08:00
Tim Abbott f3c6d91e69 mypy: Switch default to daemon mode.
There isn't any real advantage to running the non-daemon mode at this
point.
2018-12-16 21:37:58 -08:00
Tim Abbott 7485cb2a50 widgets: Rename voting_widget to poll_widget.
This ensures greater consistency with our other widgets' naming
convention.
2018-12-16 19:46:48 -08:00
Tim Abbott adebe1bd4e js: Extract csrf.js and include in common bundle.
This should make it possible to use this AJAX setup code in logged-out
code as well, which is necessary to use blueslip from portico pages.
2018-12-16 16:18:42 -08:00
Tim Abbott ed9430f3a6 Revert "js: Extract csrf.js and include in common bundle."
This reverts commit dbffa66b6d.

This broke the logged-in app, due to some webpack issue.
2018-12-16 16:16:06 -08:00
Tim Abbott dbffa66b6d js: Extract csrf.js and include in common bundle.
This should make it possible to use this AJAX setup code in logged-out
code as well, which is necessary to use blueslip from portico pages.
2018-12-16 15:35:54 -08:00
Tim Abbott 9059c4d223 provision: Make apt/yum retry logic more coherent.
This changes a few things:
* Deplicates deps_to_install logic.
* Has a retry flag, under which we can guard the apt retry print statements.
* Makes the install_system_deps flow more parallel.
2018-12-16 13:41:12 -08:00
rht c6cde1cf5c provision: Rename install_apt_deps to a more generic install_system_deps. 2018-12-16 13:39:00 -08:00
rht 389c46b4e5 provision: Extract yum-specific part of install_apt_deps. 2018-12-16 13:39:00 -08:00
rht 2be3065249 provision.py: Abstract out postgres version for CentOS. 2018-12-16 13:26:28 -08:00
rht 43bedc0909 provision: Use vendored pg_hba.conf on CentOS. 2018-12-16 13:21:54 -08:00
Tim Abbott f2173bc2a7 check-capitalization: Add a few exclude rules for new strings. 2018-12-16 11:34:31 -08:00
rht 022e926d64 provision: Setup services for CentOS. 2018-12-16 10:14:51 -08:00
rht 9c776f4fcf provision: Provide tsearch stopwords path for CentOS. 2018-12-16 10:14:08 -08:00
rht 8b5a86b992 provision: Initialize Postgres db on CentOS. 2018-12-16 10:11:55 -08:00
rht 41a5f6a49d provision: Create symlink for pg_* on CentOS. 2018-12-16 10:11:54 -08:00
rht 10786375db provision: Do not use apt-specific retry when a failure happens. 2018-12-16 10:04:27 -08:00
rht 25a8dbef11 provision: Build tsearch_extras from source for CentOS support. 2018-12-16 10:04:27 -08:00
rht 448303b3f0 provision: Initialize setup-yum-repo. 2018-12-16 10:04:27 -08:00
rht 85b2627191 provision: Add experimental support for CentOS. 2018-12-16 10:04:27 -08:00
rht 79bcb4af47 provision: Remove unused comment. 2018-12-16 10:04:27 -08:00
Steve Howell d75ff80eb2 muting: Add stream_id to markup for mute settings.
We also prefer `attr` over `data` (it's more greppable).
2018-12-14 16:05:35 -08:00
Steve Howell bf6f5e7bc5 muting: Pass stream_id to muting_ui.mute and unmute.
We temporarily allow settings_muting to have
incomplete line coverage--we will fix this soon.
2018-12-14 16:02:48 -08:00
rht 501ae0c3d6
provision: Add centos7 postgresql dependencies. 2018-12-14 04:59:48 +00:00
rht e8c602ec58
provision: Rename APT_DEPENDENCIES -> SYSTEM_DEPENDENCIES. 2018-12-14 04:59:48 +00:00
rht d54fb5f40d
provision: Add venv dependencies for Centos 7. 2018-12-14 04:59:46 +00:00
Vishnu Ks 0fd6ff722b billing: Migrate /upgrade endpoint to JSON.
The fixture changes are because self.upgrade formerly used to cause a page load
of /billing, which in turn calls Customer.retrieve.

If we ran the full test suite with GENERATE_STRIPE_FIXTURES=True, we would
likely see several more Customer.retrieve.N.json's being deleted. But
keeping them there for now to keep the diff small.
2018-12-13 17:01:12 -08:00
Anders Kaseorg 149132348f lint: Update ShellCheck to 0.6.0.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-12 08:24:01 -08:00
rht a0b0ccf972 provision: Do not require lsb_release for centos. 2018-12-11 13:08:27 -08:00
rht c9d54f7854 puppet: Remove vendored puppetlabs apt and stdlibs dependencies.
This commit works by vendoring the couple functions we still use from
puppetlabs stdlib (join and range), but removing the rest of the
puppetlabs codebase, and of course cleaning up our linter rules in the
process.

Fixes #7423.
2018-12-11 13:03:26 -08:00
Vishnu Ks a86faecff5 tools: Rename tools/travis directory to tools/ci. 2018-12-10 11:07:26 -08:00
konraddeka e6be407e34 provision: Avoid spending time on update-authors-json.
This optimizes tools/provision by not running
`tools/update-authors-json --use-fixture` unless either the script
itself or its fixtures file (zerver/tests/fixtures/authors.json) was
changed.

Fixes #10991.
2018-12-09 22:21:51 -08:00
rht 9bfea2adba provision.py: Translate COMMON_APT_DEPENDENCIES to yum. 2018-12-09 21:54:02 -08:00
Sumanth V Rao 7d12e2019d provision: Generate static bots files using a function call.
This shaves about 350ms from the no-op provision runtime.

Fixes #10990.
2018-12-09 21:44:45 -08:00
Sumanth V Rao cae718f184 provision: Rename generate_zulip_bots_static_files to .py.
This should make it possible to call this script as a function without
starting up a new Python interpreter.
2018-12-09 21:42:40 -08:00
Tim Abbott 9e48b2f17f provision: Update comment for why we need puppet in development.
This should help avoid confusion like #10972.
2018-12-06 11:13:04 -08:00
Tim Abbott 9de1bd44e2 compatibility: Clean up linter logic for too-old client. 2018-12-05 15:04:21 -08:00
Greg Price fb7bfbe9ab compatibility: Require version 16.2.96+ for our Android app.
This release is from 2018-08-22, a little over 100 days ago.

It was the first release with the important fix so that when the
server advises it to stop displaying a notification because the user
has read the message (as the SEND_REMOVE_PUSH_NOTIFICATIONS server
setting enables), the app doesn't instead replace the notification
with a broken one reading "null".  We have that setting running now
on chat.zulip.org, and intend to roll it out more broadly soon.

The `# take 0` thing is a slightly absurd workaround for the fact
that our funky out-of-line way of marking lines to ignore doesn't
work right if there are multiple such lines in a given file that
are equal modulo leading and trailing whitespace.
2018-12-05 14:58:13 -08:00
Tim Abbott 6689913f77 ci: Enable test-documentation --skip-external-links in CI.
Fixes #10942.
2018-12-05 12:39:59 -08:00
Sumanth V Rao e1f33e0f6b test-documentation: Add --skip-external-links option.
This option causes test-documentation to only verify internal links
that we control and are important to be correct.  This prevents
test-documentation from flaking in CI due to issues with the dozens of
third-party blocks that we link to from various parts of our
documentation.

Tweaked by tabbott for comment clarity, and to also include
github.com/zulip links.

Fixes #10942.
2018-12-05 12:39:59 -08:00
Tim Abbott f5f320d5fc mypy: Print out link to documentation in error output.
This should help make details about how mypy works discoverable for
developers who are new to it.
2018-12-05 11:23:55 -08:00
Tim Abbott f694d01a1b test-backend: Expand desired files with full test coverage.
This should help catch a few classes of files that we weren't
correctly considering for potentially having full test coverage.
2018-12-05 11:01:55 -08:00
Steve Howell 67fba69b0c hashchange: Go to home page for bogus narrows.
This was the original intent of the code, and I think
it's the right behavior.
2018-12-04 17:16:38 -08:00
Steve Howell bcb142e68e hashchange: Move parse_narrow to hash_util.js.
The goal here was to enforce 100% coverage on
parse_narrow, but the code has an unreachable line
and is overly tolerant of bogus urls.  This will
be fixed in the next commit.
2018-12-04 17:16:32 -08:00
Anders Kaseorg 9ba860b4f2 Avoid double redirects to /login and then to /login/.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-12-04 16:12:58 -08:00
Tim Abbott 86536059b4 tools: Fix linter tests after SCSS rename.
This was introduced in 815388b023.
2018-12-04 09:43:56 -08:00
Vishnu Ks 788b98d041 portico: Add page for redirecting to a realm subdomain. 2018-12-04 09:35:35 -08:00
Tim Abbott 815388b023 lint: Fix CSS linter rules not running.
Apparently, our CSS linter rules haven't been running since we
migrated to SCSS.
2018-12-04 09:35:34 -08:00
Vishnu Ks 6de6b0ed31 emails: Extract CSS to email.css in digest.html. 2018-12-03 17:43:53 -08:00
Tim Abbott 07adda84da Revert "ci: Re-enable testing documentation links."
This reverts commit d8e5e2d960.

See https://github.com/zulip/zulip/issues/10942 for a plan for making
this something we can re-enable.
2018-12-02 18:34:35 -08:00
Tim Abbott 1303f8df20 docs: Fix missing quotes in `su zulip -c` documentation.
This fixes an actual user-facing issue in our mobile push
notifications documentation (where we were incorrectly failing to
quote the argument to `./manage.py register_server` making it not
work), as well as preventing future similar issues from occurring
again via a linter rule.
2018-11-30 12:12:27 -08:00
Tim Abbott a3b38d8402 docs: Document nginx configuration for development SSL.
This is a useful technique for developing OAuth integrations with Zulip.
2018-11-29 20:45:02 -08:00
Steve Howell ba243416e3 stream_color: Make stream color assignment more efficient.
We now let color_data keep its own state for
unused_colors, so that we longer have to pass in
a large list of unused_colors every time we want
to assign a new stream color.

This mostly matters at startup, where we might
be cycling through 5000 streams.  We claim all
the unused colors up front.

Each operation now has an upper bound of expensiveness,
where the worst case scenario is basically popping
off the first element of a list of <= 24 colors.

The algorithm is now deterministic, too, to make
it easier to test.  It's unclear whether random color
assignment ever had much benefit, and it made unit
testing the algorithm difficult.  Now we have 100%
line coverage.

Fixes part of #10902.
2018-11-29 13:00:25 -08:00
Anders Kaseorg 0bfcd2294e tools/update-prod-static: Avoid shelling out for mkdir, cp, mv, rm.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:28:17 -08:00
Anders Kaseorg 19bdf54f33 tools/setup/emoji/build_emoji: Avoid shelling out for touch, rm, mkdir, cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-11-28 17:28:17 -08:00
Tim Abbott d8e5e2d960 ci: Re-enable testing documentation links.
We've had a few unpleasant bugs with real documentation links being
broken, so we're going to make this more aggressive for now.

I think we instead want a more subtle option for suppressing failures
in some places but not others.
2018-11-28 09:52:04 -08:00
Tim Abbott 723fb60476 docs: Fix up some broken links output spam. 2018-11-28 09:49:39 -08:00
Rishi Gupta dd0126ff1b settings: Improve error message when deactivating the last user.
This PR was originally started by Rishi Gupta (see #10383).
2018-11-27 12:49:43 -08:00
Tim Abbott adf27aae4c python: Remove now-unnecessary str_utils library.
This library was absolutely essential as part of our Python 2->3
migration process, but all of its calls should be either no-ops or
encode/decode operations.

Note also that the library has been wrong since the incorrect
refactoring in 1f9244e060.

Fixes #10807.
2018-11-27 11:57:54 -08:00
Tim Abbott 58a6dafa68 lint: Exclude outbox from linter checks on "subject". 2018-11-27 11:05:25 -08:00
Tim Abbott d5f53164b4 docs: Suppress markdown line length linter errors. 2018-11-18 18:56:44 -08:00
Steve Howell eb4d279bbc Extract get_turtle_message().
This seems like kind of a silly function to extract
to topic.py, but it will theoretically help us sweep
"subject" if we change the DB.

It had test coverage.
2018-11-14 23:24:06 -08:00
Steve Howell cc33e4cd0c digest: Eliminate unneeded queries for hot convos.
We can easily keep track of messages by bucket from the
original loop through messages.
2018-11-14 23:24:06 -08:00
Steve Howell 5a539d44b0 subject -> topic: Fix notification tests.
The URLs changed here are artificial, so we can convert
subject -> topic without breaking anything.
2018-11-14 23:24:06 -08:00
Steve Howell 27d79352da topic -> subject: Extract get_topic_from_message_info().
This changes files where it's safe to just assume caller
may use either "topic" or "subject", and we prefer "topic"
but support "subject".
2018-11-14 23:24:06 -08:00
Steve Howell 44cbcdf00c minor: Remove txt file from lint exception.
It's already ignored because it's not a .py file.
2018-11-14 23:24:06 -08:00
Steve Howell 2505a2f3cc lint: Build exceptions for subject. 2018-11-14 23:24:06 -08:00
Steve Howell a3460d1ef6 lint: Remove obsolete exemptions.
These two files have been cleaned up since the original
lint rule exemption.
2018-11-14 23:24:06 -08:00
Steve Howell c69c40a55a lint: Extract check_file_for_pattern(). 2018-11-14 23:24:06 -08:00
Steve Howell fd0b249a29 lint: Calculate strip_rule outside the loop.
This adds up over a million lines. :)
2018-11-14 23:24:06 -08:00
Steve Howell f386c64759 lint: Speed up by doing fewer regex compiles.
Even though you'd think these regexes would be
cached, compiling the regex outside of looping
through lines makes a difference.

My timings are 8.4s -> 6.0s.  (You need to hack
on the linter to isolate the custom checks.)
2018-11-14 23:24:06 -08:00
Steve Howell 08e44aac0d lint: Extract get_rules_applying_to_fn(). 2018-11-14 23:24:06 -08:00
Steve Howell eda138343b lint: Extract get_line_info_from_file(). 2018-11-14 23:24:06 -08:00
Steve Howell 6d88210110 lint: Extract check_file_for_long_lines().
In passing this removes some unnecessary string
decoding that's a relic of python2.
2018-11-14 23:24:06 -08:00
Steve Howell a942732d83 lint: Get rid of unused skip_rules.
We also go to the modern annotation style for mypy.
2018-11-12 16:00:52 -08:00
Steve Howell 3eb2b2be5f lint: Change lastLine logic slightly. 2018-11-12 15:47:11 -08:00
Steve Howell a86e7fbd9f subject -> topic: Fix message tests. 2018-11-12 15:47:11 -08:00
Steve Howell c90085b28e lint: Extend "subject" prohibition to zerver/tests.
There are several exemptions here, so I triaged
them into various buckets.
2018-11-12 15:47:11 -08:00
Steve Howell d86dd165da gitter/slack/hipchat: Remove "subject" from conversions.
We (lexically) remove "subject" from the conversion code.  The
`build_message` helper calls `set_topic_name` under the hood,
so things still have "subject" in the JSON.

There was good code coverage on `build_message`.
2018-11-12 15:47:11 -08:00
Steve Howell ea98a44db3 webhooks: Replace SUBJECT_WITH_* with TOPIC_WITH_*. 2018-11-12 15:47:11 -08:00
Steve Howell 97062c4017 lint: Prevent use of "subject" in zerver/views. 2018-11-12 15:47:11 -08:00
Eeshan Garg 10570d508f tools/lint: Fix JSON lint pattern for spaces after ':'.
Previously, a string ending in "... 😄" was reported as an
error and the linter complained that there should be a space
after the last ':'. This commit changes the pattern so that the
linter only checks for colons that are preceded by an opening
double-quote (").
2018-11-08 22:07:56 -08:00
Steve Howell cd4d447d90 Minor: change message.subject -> message.topic_name().
This also allows us to lock down
push_notifications.py from using "subject"
in the future.
2018-11-08 16:34:30 +00:00
Steve Howell ff9a6c5ced minor: Rename subject -> topic_name in bugdown. 2018-11-08 16:21:14 +00:00
Steve Howell 70c16ddbcc lint: Prevent "subject" more aggressively.
We now prevent adding "subject" to any code in
zerver/lib, unless you specifically exempt it.

The new set called `FILES_WITH_LEGACY_SUJECT`
is also has comments that give a roadmap of what
to fix.
2018-11-08 15:33:36 +00:00
Steve Howell e802be0fde Remove "subject" in notifications.py.
These last few are all pretty safe.
2018-11-08 15:12:10 +00:00
Steve Howell 226a2826b5 Remove "subject" from zerver/lib/message.py.
We also lock this down file in the linter, so
that you can no longer introduce "subject"
to `message.py`.
2018-11-08 14:45:16 +00:00
Steve Howell 6b6001c46c Remove "subject" from test_classes.py.
We now use "topic" lingo.
2018-11-07 10:03:53 -08:00
Steve Howell 79d5e36ca3 Extract topic_match_sa() helper.
We'll also use this in zerver/views/messages.py, but
that's a bigger change.
2018-11-07 10:03:53 -08:00
Steve Howell df743e8948 lint: Start to deprecate "subject" references. 2018-11-07 10:03:53 -08:00
Rishi Gupta 69e9a8f196 user docs: Add a few portico URLs to documentation crawler test. 2018-11-01 11:59:50 -07:00
Tim Abbott d9887759b4 plans: Fix broken link from FAQ to import docs and add tests. 2018-10-31 11:44:21 -07:00
zegervdv 6dd2218c50 provision: Pass proxy settings through sudo via environment.
This makes provision not fail when installing node behind a proxy.

See #10505 for details.
2018-10-29 13:30:45 -07:00
Vishnu Ks 0cf4a75fb5 coverage: Enforce full test coverage for corporate/views.py. 2018-10-24 19:39:23 -07:00
Anders Kaseorg c3bd293eaf Add shellcheck as a linter.
Fixes #10162.

Tweaked by tabbott to cover documentation as well.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-18 15:32:26 -07:00
Anders Kaseorg 49ae9c1e44 pre-commit: Fix shellcheck warnings.
In tools/pre-commit line 18:
if [ -z "$VIRTUAL_ENV" ] && `which vagrant > /dev/null` && [ -e .vagrant ]; then
                            ^-- SC2092: Remove backticks to avoid executing output.
                            ^-- SC2006: Use $(..) instead of legacy `..`.
                             ^-- SC2230: which is non-standard. Use builtin 'command -v' instead.

In tools/pre-commit line 23:
    ./tools/lint --no-gitlint --force $changed_files || true
                                      ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-18 15:26:21 -07:00
Anders Kaseorg fc24d2e147 install-aws-server: Fix shellcheck warnings.
In tools/setup/install-aws-server line 25:
zulip_root=${ZULIP_ROOT:-$HOME/zulip}
^-- SC2034: zulip_root appears unused. Verify use (or export if used externally).

In tools/setup/install-aws-server line 40:
if [ -n "$zulip_confdir" ]; then
         ^-- SC2154: zulip_confdir is referenced but not assigned.

In tools/setup/install-aws-server line 55:
VIRTUALENV_NEEDED=$(if $(echo "$type" | grep -q app_frontend); then echo -n yes; else echo -n no; fi)
                       ^-- SC2091: Remove surrounding $() to avoid executing output.

In tools/setup/install-aws-server line 60:
SSH_OPTS=(-o HostKeyAlgorithms=ssh-rsa)
             ^-- SC2191: The = here is literal. To assign by index, use ( [index]=value ) with no spaces. To keep as literal, quote it.

In tools/setup/install-aws-server line 69:
ssh "${SSH_OPTS[@]}" "$server" -t -i "$amazon_key_file" -lroot <<EOF
                                                                 ^-- SC2087: Quote 'EOF' to make here document expansions happen on the server side rather than on the client.

In tools/setup/install-aws-server line 86:
ssh "${SSH_OPTS[@]}" "$server" -t -i "$amazon_key_file" -lroot <<EOF
                                                                 ^-- SC2087: Quote 'EOF' to make here document expansions happen on the server side rather than on the client.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:51:41 -07:00
Anders Kaseorg b15063155c postgres-init-dev-db: Fix shellcheck warnings.
In tools/setup/postgres-init-dev-db line 10:
ROOT_POSTGRES="sudo -i -u "$DEFAULT_USER" psql"
                           ^-- SC2027: The surrounding quotes actually unquote this. Remove or escape them.

In tools/setup/postgres-init-dev-db line 46:
    echo 'ERROR: Try `sudo service postgresql start`?'
         ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

In tools/setup/postgres-init-dev-db line 64:
PGPASS_ESCAPED_PREFIX="*:\*:\*:$USERNAME:"
                         ^-- SC1117: Backslash is literal in "\*". Prefer explicit escaping: "\\*".
                            ^-- SC1117: Backslash is literal in "\*". Prefer explicit escaping: "\\*".

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:48:57 -07:00
Anders Kaseorg 7f35ad916e django-template-graph: Fix shellcheck warnings.
In tools/django-template-graph line 10:
    for t in $(find -name '*.html' -printf '%P\n'); do
             ^-- SC2044: For loops over find output are fragile. Use find -exec or a while read loop.
               ^-- SC2185: Some finds don't have a default path. Specify '.' explicitly.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg c63d852b55 deploy-branch: Fix shellcheck warnings.
In tools/deploy-branch line 17:
[ $? -ne 0 ] && error_out "Unknown branch: $branch"
  ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In tools/deploy-branch line 23:
    if [ $? -eq 0 ]; then
         ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In tools/deploy-branch line 35:
[ $? -ne 0 ] && error_out "Rebase onto origin/master failed"
  ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

In tools/deploy-branch line 39:
[ $? -ne 0 ] && error_out "Push of master to origin/master failed"
  ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg 4a8074000c commit-msg: Fix shellcheck warnings.
In tools/commit-msg line 9:
if [ $(grep '^[^#]' .git/COMMIT_EDITMSG --count) -ne 0 ]; then
     ^-- SC2046: Quote this to prevent word splitting.

In tools/commit-msg line 10:
    lint_cmd="cd ~/zulip && cat \"$1\" | python -m gitlint.cli"
              ^-- SC2089: Quotes/backslashes will be treated literally. Use an array.

In tools/commit-msg line 11:
    if [ -z "$VIRTUAL_ENV" ] && `which vagrant > /dev/null` && [ -e .vagrant ]; then
                                ^-- SC2092: Remove backticks to avoid executing output.
                                ^-- SC2006: Use $(..) instead of legacy `..`.
                                 ^-- SC2230: which is non-standard. Use builtin 'command -v' instead.

In tools/commit-msg line 14:
        $lint_cmd
        ^-- SC2090: Quotes/backslashes in this variable will not be respected.

In tools/commit-msg line 17:
    if [ $? -ne 0 ]; then
         ^-- SC2181: Check exit code directly with e.g. 'if mycmd;', not indirectly with $?.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg 8b38f61176 clean-branches: Fix shellcheck warnings.
In tools/clean-branches line 33:
                echo -n "Deleting local branch $(echo "$ref" | sed 's!^refs/heads/!!')"
                                                 ^-- SC2001: See if you can use ${variable//search/replace} instead.

In tools/clean-branches line 41:
                echo -n "Deleting local branch $(echo "$ref" | sed 's!^refs/heads/!!')"
                                                 ^-- SC2001: See if you can use ${variable//search/replace} instead.

In tools/clean-branches line 49:
                remote_name="$(echo "$ref" | sed 's!^refs/remotes/origin/!!')"
                               ^-- SC2001: See if you can use ${variable//search/replace} instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg b5bfe77844 build-release-tarball: Fix shellcheck warnings.
In tools/build-release-tarball line 50:
for i in `cat "$TMPDIR/$prefix/tools/release-tarball-exclude.txt"`; do
         ^-- SC2013: To read lines rather than words, pipe/redirect to a 'while read' loop.
         ^-- SC2006: Use $(..) instead of legacy `..`.

In tools/build-release-tarball line 51:
    rm -r --interactive=never "$TMPDIR/$prefix/$i";
                              ^-- SC2115: Use "${var:?}" to ensure this never expands to / .

In tools/build-release-tarball line 97:
    echo; echo -ne "\033[33mRunning update-prod-static failed. "
                    ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/build-release-tarball line 98:
    echo -e "Check $TMPDIR/update-prod-static.log for more information.\033[0m"
                                                                       ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Anders Kaseorg a5280d7d02 build-docs: Fix shellcheck warnings.
In tools/build-docs line 3:
cd "$(dirname "$0")"/../docs
^-- SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-10-17 17:38:56 -07:00
Rishi Gupta d76f728dda api docs: Rename webhook-walkthrough to incoming-webhooks-walkthrough. 2018-10-16 21:07:00 -07:00
Eeshan Garg 0292ffe04b webhooks/taiga: Add test for when a task is moved between stories.
This commit adds a test for the payload that is generated when
a Task is moved from one user story to another on Taiga's Sprint
Taskboard UI.

This commit also gets up this webhook's test coverage up to 100%.
2018-10-16 13:06:20 -07:00
Aditya Bansal 75ae94e459 font-awesome: Drop support for legacy font awesome icons.
We drop support for usage of `icon-vector` as base class when
including icons from font awesome icons package.
Now on, only icons as specified in font awesome v4.7.0 can be used
in the code base.
2018-10-15 20:14:55 +05:30
Aditya Bansal 7eeed1413c icons: Remove base class from comments and test templates. 2018-10-15 19:55:18 +05:30
Steve Howell b7f764aa29 settings: Add confirm_dialog module.
This module makes it really easy to create are-you-sure
dialogs for dangerous operations.

Basically it's one function with five parameters.  You
give three chunks of HTML, a callback function, and
a parent container.

The first use of this will be in settings_user_groups,
coming up in a couple commits.
2018-10-12 10:37:06 -07:00
Eeshan Garg ba929508e2 webhooks/ifttt: Get test coverage up to 100%.
IFTTT allows custom templating for their payloads, so the onus is
on the user to ensure that their custom templates conform to the
expectations outlined in our IFTTT webhook docs. For that reason,
these payloads weren't generated, but were manually edited.
2018-10-04 12:16:06 -07:00
Eeshan Garg 316f9aa78b webhooks/hellosign: Rewrite the integration from scratch.
After discovering a couple of bugs, I decided to thoroughly test
and rewrite this integration from scratch. The older code wasn't
generating coherent messages.

This also commit gets this integration up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg 5039f6dfb5 webhooks/gitlab: Get test coverage up to 100%.
Test coverage was improved by removing an unused function and
removing some code (written by me) that was actually handling
Test Hook event types incorrectly.
2018-10-04 12:16:06 -07:00
Eeshan Garg 6e2e2b9125 webhooks/github: Test commit status payloads with target_url.
It was a painful amount of work to generate the actual payload.
Since the only difference was a small build URL, I manually
edited the payload and used that for testing.

This commit gets our GitHub webhook up to 100% test coverage.
2018-10-04 12:16:06 -07:00
Eeshan Garg b40aec3a09 webhooks/freshdesk: Improve test coverage.
Note that Freshdesk allows custom templating for outgoing payloads
in their webhook UI. Therefore, the payloads added in this commit
did not have to be official payloads from Freshdesk.
2018-10-04 12:16:06 -07:00
Eeshan Garg d8101ca139 webhooks/bitbucket2: Improve test coverage.
The lack of coverage was due to:

* An unused function that was never used anywhere.
* get_commit_status_changed_body was using a regex where it didn't
  really need to use one. And there was an if statement that
  assumed that the payload might NOT contain the URL to the commit.
  However, I checked the payload and there shouldn't be any instances
  where a commit event is generated but there is no URL to the commit.
* get_push_tag_body had an `else` condition that really can't happen
  in any payload. I verified this by checking the BitBucket webhook
  docs.
2018-10-04 12:16:06 -07:00
Eeshan Garg 185a023745 webhooks/beanstalk: Improve test coverage.
We shouldn't just ignore exceptions when encoding the incoming
auth credentials. Even if the incoming credentials are properly
encoded, it is better to know when that is the case or if
something else fails.
2018-10-04 12:16:06 -07:00
Harshit Bansal fc7e654d7b emoji: Remove single-digit names for number emojis.
Names like `:1:`, `:2:` etc don't seem to be of much benefit rather
they seem to get rendered at places not intended.
2018-10-02 11:30:11 -07:00
Vishnu Ks 59865797ac billing: Move billing tests from zilencer to corporate. 2018-10-01 15:44:02 -07:00
Vishnu Ks 93b398bc0a billing: Move zilencer/lib/stripe to corporate. 2018-10-01 15:43:44 -07:00
Vishnu Ks daf3b46262 billing: Move billing related templates from zilencer to corporate. 2018-10-01 15:43:39 -07:00
Vishnu Ks 6914ee126c billing: Move billing-related views and urls to corporate. 2018-10-01 15:43:21 -07:00
Tim Abbott 00821d67bb travis: Remove no-longer-present packages from apt-mark hold list.
Apparently, Travis removed the Heroku bundle of packages from their
servers, which made the build start failing when trying to configure
apt to hold their versions (sigh).  This commit removes the
problematic packages.
2018-09-28 14:20:04 -07:00
Greg Price 3b646c5d28 push-to-pull-request: Update local tracking ref, if any.
The companion tool `tools/reset-to-pull-request` has a handy feature
to maintain a local ref tracking the PR: e.g., pr/1234 for PR 1234.
If this were a normal remote-tracking branch maintained by `git fetch`,
it'd get updated on `git push`.  Do the same thing here.

This helps keep a view like `gitk --all @` a bit tidier, by causing
merged PRs to stop pointing at side branches of the main history.
2018-09-25 13:15:02 -07:00
Vishnu Ks 5a6b2ebb1f billing: Add ability for users to change their card. 2018-09-24 19:25:26 -07:00
Anders Kaseorg 5e88b3013a zanitizer_config.pm.sample: keep_file → scrub_filename
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-09-23 20:42:27 -07:00
Eeshan Garg ecd4f821be user docs: Automatically activate correct tab for OS-specific instructions. 2018-09-18 13:49:34 -07:00
Vishnu Ks d6b548bd57 styles: Extract billing SCSS into its own stylesheet. 2018-09-11 00:08:38 -07:00
Steve Howell 06225d1424 tests: Clean up calls to tools/webpack.
Before this change, the way we loaded
webpack for various tools was brittle.

First, I addressed test-api and test-help-documentation.

These tools used to be unable to run standalone on a
clean provision, because they were (indirectly)
calling tools/webpack without the `--test` option.

The problem was a bit obscure, since running things
like `./tools/test-backend` or `./tools/test-all` in
your workflow would create `./var/webpack-stats-test.json`
for the broken tools (and then they would work).

The tools themselves weren't broken; they were the
only relying on the common `test_server_running` helper.
And even that helper wasn't broken; it was just that
`run-dev.py` wasn't respecting the `--test` option.

So I made it so that `./tools/run-dev` passes in `--test` to
`./tools/webpack`.

To confuse matters even more, for some reason Casper
uses `./webpack-stats-production.json` via various
hacks for its webpack configuration, so when I fixed
the other tests, it broke Casper.

Here is the Casper-related hack in zproject/test_settings.py,
which was in place before my change and remains
after it:

    if CASPER_TESTS:
        WEBPACK_FILE = 'webpack-stats-production.json'
    else:
        WEBPACK_FILE = os.path.join('var', 'webpack-stats-test.json')

I added similar logic in tools/webpack:

    if "CASPER_TESTS" in os.environ:
        build_for_prod_or_casper(args.quiet)

I also made the helper functions in `./tools/webpack` have
nicer names.

So, now tools should all be able to run standalone and not
rely on previous tools creating webpack stats files for
them and leaving them in the file system.  That's good.

Things are still a bit janky, though.  It's not completely
clear to me why `test-js-with-casper` should work off of
a different webpack configuration than the other tests.

For now most of the jankiness is around Casper, and we have
hacks in two different places, `zproject/test_settings.py` and
`tools/webpack` to force it to use the production stats
file instead of the "test" one, even though Casper uses
test-like settings for other things like which database
you're using.
2018-09-07 11:39:55 -04:00
Steve Howell 25ad755914 Fix usage for test-api. 2018-09-07 11:39:55 -04:00
Steve Howell 336c2cee20 Revert unused code for creating non-editable pills.
We don't use input.create_non_editable_pill() in our
code yet.  If we add this back, we'll want to have node
tests on it.

Removing this unused code brings us to 100% line
coverage for input_pill.js.

This directly reverts 5c11ab85 with the small addition
of adding input_pill to our list of fully covered
modules.
2018-08-31 16:26:05 -07:00
Harshit Bansal 88bfb9778b build_emoji: Add infrastructure for google-blob and twitter emojisets. 2018-08-31 11:52:24 -07:00
Marco Burstein 25e624eab1 team: Include contributions from legacy repositories in the total.
These repositories (`zulip-ios-legacy` and `zulip-android`) are
deprecated, and as such should not have their own tabs, but still
should be included in the total contributions count.
2018-08-28 16:45:00 -07:00
Tim Abbott 2c7f9ce0fc puppet: Fix puppet-lint warnings in various manifests.
Apparently, `puppet-lint` on Ubuntu trusty throws warnings for certain
quoting patterns that are OK in modern `puppet-lint`.  I believe the
old Zulip code was actually correct (i.e. the old `puppet-lint`
implementation was the problem), but it seems worth changing anyway to
suppress the warnings.

We also exclude more of puppet-apt from linting, since it's
third-party code.
2018-08-28 13:46:31 -07:00
Lyla Fischer 1efcdfdb5c templates: Merge Help and API doc main into documentation_main.
With minor fixes by eeshangarg.
2018-08-27 20:19:24 -02:30
Harshit Bansal e204fe3948 minor: Fix for failing tests. 2018-08-27 06:16:16 +00:00
Harshit Bansal d96d6a96f7 build_emoji: Add a missing check for image existence. 2018-08-26 22:16:23 -07:00
Harshit Bansal f5454dff51 build_emoji: Automatically calculate sprite dimensions.
Instead of using a hardcoded value for spritesheet dimensions,
automatically calculate it using `emoji_data`. This will free
us from updating it only emoji datasource update as well as
allow us to add google blob emojiset.
2018-08-26 22:16:23 -07:00
Harshit Bansal dbc3774c2f build_emoji: Refactor `get_emoji_code()` to not fail for old datasources.
If `non_qualified` field is not present then return the value of
`unified` field instead of failing.
2018-08-26 22:16:23 -07:00
Harshit Bansal ec2ebd1cc9 build_emoji: Change sprite CSS files to use hyphens in names.
This is more consistent with our other names in emoji infra.
2018-08-26 22:16:23 -07:00
Harshit Bansal 9bd56d0ce4 build_emoji: Change emoji sprite sheets to use hyphens in names.
Use of hyphens is more consistent with our other emoji infra names.
2018-08-26 22:16:23 -07:00
Harshit Bansal aa6f098496 build_emoji: Extract `setup_emoji_farm()`. 2018-08-26 22:16:23 -07:00
Harshit Bansal dc9ff928f3 build_emoji: Refactor `ensure_emoji_image()` to take paths as args. 2018-08-26 22:16:23 -07:00
Harshit Bansal ed694001da build_emoji: Rename `setup_emoji_farm()` to `setup_emoji_farms()`. 2018-08-26 22:16:23 -07:00
Steve Howell ca272ef49d tests: Test how we handle extra data for zform messages. 2018-08-24 10:00:25 -07:00
Steve Howell bee592b44e node tests: Bring list_cursor to 100% coverage.
We had some indirect coverage of this already, so
this mostly focuses on error cases and corner
cases.
2018-08-24 10:00:04 -07:00
Steve Howell 44756a326d tests: Test duplicate default stream/groups.
This also gets us to 100% line coverage
for actions.py.
2018-08-22 16:52:30 -07:00
Steve Howell 4318f75718 Remove `code` from JsonableError.__init__.
Since this class was built, folks have always chosen
to subclass JsonableError for situations where
the default of ErrorCode.BAD_REQUEST is insufficient.

So now we simplify the use cases, which also gets
us 100% coverage on this core module.
2018-08-22 16:51:40 -07:00
Tim Abbott b93d5bd5e8 test-js-with-node: Temporarily remove server_events_dispatch.js.
This fixes master failing tests since 2aeabf24a6.
2018-08-21 14:40:05 -07:00
Joshua Pan 793ec9e266 tests: Test all update_display_settings events. 2018-08-19 14:51:31 -04:00
xificurC 9e053c74cf scripts: Allow configuring a custom CA bundle for build process.
For building Zulip in an environment where a custom CA certificate is
required to access the public Internet, one needs to be able to
specify that CA certificate for all network access done by the Zulip
installer/build process.  This change allows configuring that via the
environment.
2018-08-13 15:45:28 -07:00
Tim Abbott 90828297e4 puppet-lint: Enforce double_quoted_strings check.
This makes our puppet codebase more consistent by using single-quoted
strings consistently.
2018-08-13 12:31:19 -07:00
Tim Abbott d0b51b70f4 puppet-lint: Enforce 2sp_soft_tables puppet-lint check.
This cleans up the puppet codebase's whitespace formatting to be more
consistent.
2018-08-13 12:31:16 -07:00
Tim Abbott b26e0a957d puppet-lint: Enforce arrow_alignment check.
This fixes all exceptions in our puppet codebase to this lint rule.
2018-08-13 12:30:57 -07:00
Aditya Bansal a3b8ab46b3 puppet-lint: Include in linter tool chain to run by default.
We start to use puppet-lint to lint puppet modules by default by
adding it to tools/lint (which controls our linter tool chain).

We also define a few puppet-lint rules to exclude.

Fixes: #9185.
2018-08-13 12:30:57 -07:00
Aditya Bansal 56d49980de puppet-lint: Add as a dependancy to install during provision process. 2018-08-13 12:30:57 -07:00
Harshit Bansal 29daf76420 emoji: Don't generate emoji farms corresponding to now banned emojisets.
This will help us in reducing the size of the release tarball
significantly. I have refrained from changing the `EMOJISETS`
constant in the `emoji_setup_utils.py` as that controls the
emojisets that we want to support. Since we want to re-enable
the feature of changing emojisets sometime again in the future
that variable should be kept as it is as it controls several
other things like emoji scripts that we use to generate emoji
names. Changing it might cause hard to catch bugs.
2018-08-13 12:09:06 -07:00
Harshit Bansal 36b977c382 emoji: Refactor `generate_sprite_css_files()` to take emojiset in args. 2018-08-13 12:09:06 -07:00
Harshit Bansal 67bbbbb007 emoji: Move logic for hard-coding octopus emoji.
This logic was incorrectly placed in `setup_old_emoji_farm()` while
it should have been placed in `setup_emoji_farm()`.
2018-08-13 12:09:06 -07:00
Harshit Bansal 1eb3825048 emoji: Remove now-unnecessary `get_remapped_emojis_map()`. 2018-08-13 12:08:14 -07:00
Harshit Bansal 99e1a81055 emoji: Fix some emoji images not loading in missed message emails.
`emoji-datasource` package v4.0.4 introduced the concept of qualified
and non-qualified emoji codes. As chat programs don't need to use
emoji representation selector, so we used migrated our infrastructure
to use non-qualified emoji codes. But we missed the fact that the
emoji file names in emoji farm are based on emoji data's 'unified'
field and the value of this field has changed. Consequently the image
file names must also have been changed. We used `emoji_code` while
converting the span tags to img tags while processing notifications.
But since now `emoji_code` refers to non-qualified code while image
file names are based on qualified code, we need to rename images
to correctly do the conversion. This commit just fixes this.
2018-08-13 12:03:16 -07:00
Steve Howell 413a0174f4 Extract a zephyr.py library.
Right now it only has one function, but the function
we removed never really belonged in actions.py, and
now we have better test coverage on actions.py, which
is an important module to get to 100%.
2018-08-11 14:51:26 -07:00
Max Nussenbaum 570b9515c6 billing: Add location for errors on billing page.
This adds a location for error messages to the billing page,
using the standard error styling.
2018-08-10 22:48:10 -07:00
Vishnu Ks 0c84260faa billing: Enforce full coverage in zilencer/tests. 2018-08-10 13:52:35 -07:00
Vishnu Ks 9946202148 coverage: Enforce full coverage for zilencer/lib/stripe.py. 2018-08-10 13:52:35 -07:00
Tim Abbott aaedec1fdb tools: Move a few i18n scripts to tools/i18n.
These are likely to only be run by Zulip maintainers.
2018-08-09 15:46:27 -07:00
Tim Abbott ba626dcad6 tools: Move optimize-svg to tools/setup/. 2018-08-09 15:39:57 -07:00
Tim Abbott c2e5cc99a4 tools: Move generate-custom-icon-webfont to a subdirectory. 2018-08-09 15:39:57 -07:00
Tim Abbott 2bb04fc720 tools: Remove send-github-payloads.
This ancient tool predates our practice of collecting test fixtures
for third-party integrations, which is a better general system for the
problem this solved.
2018-08-09 15:39:57 -07:00
Tim Abbott 74dc3e9a3e tools: Remove unused munge-postgres-logs.
This hasn't been used in several years.
2018-08-09 15:39:57 -07:00
Tim Abbott bd945333a3 tools: Remove replace-tarball-shebang.
This was a temporary helper tool for part of
tools/build-release-tarball during our Python 3 migration, which
finished some time ago.
2018-08-09 15:27:09 -07:00
Tim Abbott 78539e7819 tools: Remove unused deployment-lock-ctl.
This interacted with a deployment locking system that is no longer
part of the project.
2018-08-09 15:25:15 -07:00
Rhea Parekh 26300110ca import: Fix rendered_content in imported messages.
After the messages have been imported, set the rendered_content of the
messages instead of leaving its value to be 'None'.

This is important to ensure that:
(1) Performance for users is good after completing the import.
(2) The database's full-text indexes have all of the imported messages
(which only happens properly when Message rows have their
rendered_content field edited).

Fixes #9168.
2018-08-09 15:12:53 -07:00
Yago González f6219745de users: Get all API keys via wrapper method.
Now reading API keys from a user is done with the get_api_key wrapper
method, rather than directly fetching it from the user object.

Also, every place where an action should be done for each API key is now
using get_all_api_keys. This method returns for the moment a single-item
list, containing the specified user's API key.

This commit is the first step towards allowing users have multiple API
keys.
2018-08-08 16:35:17 -07:00
Yago González a0d8cbc9cb test-backend: Fix typo. 2018-08-08 16:34:29 -07:00
Cynthia Lin ecbc58c586 settings: Fix color styling for nested inputs to work in night mode. 2018-08-08 11:29:48 -07:00
Rhea Parekh 30cc7354eb import: Move 'process_avatars' to import_util. 2018-08-07 16:45:40 -07:00
Rhea Parekh 1117455a90 import: Move 'ZerverFieldsT' and 'build_zerver_realm' to import_util. 2018-08-07 16:35:56 -07:00
Vishnu Ks 119d108da0 droplets: Update code and remove nvm settings from bashrc.
* Synced zulip repo to master and provisioned
* Synced python-zulip-api repo to master and provisioned.
* Removed unnecessary nvm config from bashrc
2018-08-07 13:38:18 -07:00
Tim Abbott 4300683bca lint: Fix mypy error in check_pyflakes annotations.
Introduced in 78a93b8d9e.
2018-08-07 09:59:14 -07:00
Tim Abbott e67fa36803 lint: Update config to split files more evenly. 2018-08-07 09:42:37 -07:00
Tim Abbott 78a93b8d9e lint: Refactor pyflakes to use more consistent style.
In particular, we no longer pass in the full `by_lang` object and
expect it to pull out the Python piece.
2018-08-07 09:42:37 -07:00
Tim Abbott 458c17868f zulint: Move pycodestyle runner code to library.
This way, only the Zulip project's specific configuration ends up
living in pep8.py.
2018-08-04 19:54:18 -07:00
Tim Abbott 3b53ddf779 zulint: Stop importing color codes from zulip_tools.py.
This is I think the final dependency that zulint/ has on the rest of
the project.
2018-08-04 19:53:53 -07:00
Tim Abbott c0beeebc3e zulint: Move printer.py module to tools/zulint.
At this point, tools/linter_lib is intended to be the directory of
Zulip-specific code.
2018-08-04 19:53:53 -07:00
Tim Abbott 55ba434ecb zulint: Extract LinterConfig class to zulint. 2018-08-04 19:53:53 -07:00
Tim Abbott f74bb8a9b2 zulint: Extract run_parallel and do_lint. 2018-08-04 19:53:53 -07:00
Tim Abbott acf8ec492d zulint: Extract common linter argument parsing logic.
The remaining code is Zulip-specific business logic.
2018-08-04 19:53:53 -07:00
Tim Abbott 1fcc2a6ea4 zulint: Move lister.py to tools/zulint.
This is preparatory refactoring work for being able to extract the
linter as an external project called "zulint".
2018-08-04 19:53:53 -07:00
Joshua Pan 5303281dfb tests: Fully cover hash_util.js. 2018-08-04 17:56:02 -07:00
Abhilash Verma fd9652a9d7 settings_emoji: Make strings more user-friendly. 2018-08-04 09:37:48 -07:00
Steve Howell c8898e1dc8 refactor: Move by_conversation_and_time_uri to hash_util.
This removes the 100% coverage on hash_util, but we are
pretty careful about not messing with this code.
2018-08-04 09:32:27 -07:00
Anders Kaseorg d79555e9be update-locked-requirements: Fix shellcheck warnings.
In tools/update-locked-requirements line 66:
compile_requirements requirements/prod.in $OUTPUT_BASE_DIR/prod.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 67:
compile_requirements requirements/dev.in $OUTPUT_BASE_DIR/dev.txt
                                         ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 68:
compile_requirements requirements/mypy.in $OUTPUT_BASE_DIR/mypy.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 69:
compile_requirements requirements/docs.in $OUTPUT_BASE_DIR/docs.txt
                                          ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/update-locked-requirements line 70:
compile_requirements requirements/thumbor.in $OUTPUT_BASE_DIR/thumbor.txt py2
                                             ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg 3e7bf9be9c production-helper: Fix shellcheck warnings.
In tools/travis/production-helper line 24:
if ! apt-get dist-upgrade -y $APT_OPTIONS; then
                             ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/travis/production-helper line 26:
    apt-get dist-upgrade -y $APT_OPTIONS
                            ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e05a56aa8b test-migrations: Fix shellcheck warnings.
In tools/test-migrations line 18:
    echo "$new_auto_named_migrations" | sed 's/\[[x ]\] /  /'
    ^-- SC2001: See if you can use ${variable//search/replace} instead.

In tools/test-migrations line 27:
    echo 'ERROR: Migrations are not consistent with models!  Fix with `./tools/renumber-migrations`.'
         ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c4ed4bc021 test-install: Fix shellcheck warnings.
In tools/test-install/destroy-all line 31:
  | while read c
          ^-- SC2162: read without -r will mangle backslashes.

In tools/test-install/install line 57:
    installer_dir="$(readlink -f $INSTALLER)"
                                 ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/test-install/lxc-wait line 30:
for i in {1..60}; do
^-- SC2034: i appears unused. Verify use (or export if used externally).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e1d770d451 test-documentation: Fix shellcheck warnings.
In tools/test-documentation line 6:
    echo -e "\e[${color_code}m${message}\e[0m" >&2
             ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                        ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".

In tools/test-documentation line 41:
scrapy crawl_with_status documentation_crawler $loglevel
                                               ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg bb51503907 test-all-docker: Fix shellcheck warnings.
In tools/test-all-docker line 7:
source /home/zulip/.bash_profile
^-- SC1091: Not following: /home/zulip/.bash_profile: openBinaryFile: does not exist (No such file or directory)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c93f53b463 test-all: Fix shellcheck warnings.
In tools/test-all line 7:
TEMP=`getopt -o f --long force -- "$@"`
     ^-- SC2006: Use $(..) instead of legacy `..`.

In tools/test-all line 24:
    echo "Running $@"
                  ^-- SC2145: Argument mixes string and array. Use * or separate argument.

In tools/test-all line 26:
        printf "\n\e[31;1mFAILED\e[0m $@\n"
               ^-- SC2059: Don't use variables in the printf format string. Use printf "..%s.." "$foo".
                ^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\\n".
                  ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                ^-- SC1117: Backslash is literal in "\e". Prefer explicit escaping: "\\e".
                                      ^-- SC2145: Argument mixes string and array. Use * or separate argument.
                                        ^-- SC1117: Backslash is literal in "\n". Prefer explicit escaping: "\\n".

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg e9656564c2 start-dockers: Fix shellcheck warnings.
In tools/start-dockers line 7:
source /home/zulip/.bash_profile
^-- SC1091: Not following: /home/zulip/.bash_profile: openBinaryFile: does not exist (No such file or directory)

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg f4c4e857ff generate-test-credentials: Fix shellcheck warnings.
In tools/setup/generate-test-credentials line 10:
password=$(./manage.py print_initial_password "$email" | fgrep "$email" | awk '{ print $2 }')
                                                         ^-- SC2197: fgrep is non-standard and deprecated. Use grep -F instead.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg af64bff9d4 reset-to-pull-request: Fix shellcheck warnings.
In tools/reset-to-pull-request line 25:
    git fetch "$remote" +"pull/$request_id/head":"$target_ref"
                                                ^-- SC2140: Word is of the form "A"B"C" (B indicated). Did you mean "ABC" or "A\"B\"C"?

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg c669626666 provision: Fix shellcheck warnings.
In tools/provision line 13:
FAIL="\033[91m"
      ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 14:
WARNING="\033[93m"
         ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 15:
ENDC="\033[0m"
      ^-- SC1117: Backslash is literal in "\0". Prefer explicit escaping: "\\0".

In tools/provision line 19:
PARENT_PATH=$( cd "$(dirname "${BASH_SOURCE}")" ; pwd -P )
                              ^-- SC2128: Expanding an array without an index only gives the first element.

In tools/provision line 32:
if [ $failed = 1 ]; then
     ^-- SC2086: Double quote to prevent globbing and word splitting.

In tools/provision line 49:
    echo 'or just close this shell and start a new one (with Vagrant, `vagrant ssh`).'
         ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg d7c7e8dd7c optimize-svg: Fix shellcheck warnings.
In tools/optimize-svg line 3:
if [ `node_modules/.bin/svgo -f static/images/integrations/logos | grep -o '\.[0-9]% = ' | wc -l` -ge 1 ]
     ^-- SC2046: Quote this to prevent word splitting.
     ^-- SC2006: Use $(..) instead of legacy `..`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:27 -07:00
Anders Kaseorg cd451491ff find-unused-css: Fix shellcheck warnings.
In tools/find-unused-css line 5:
    if [ $(git grep "$n" | grep -v '^static/styles/zulip.css' | wc -l) -eq 0 ]; then
         ^-- SC2046: Quote this to prevent word splitting.
                           ^-- SC2126: Consider using grep -c instead of grep|wc -l.

In tools/find-unused-css line 6:
        echo $n
             ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg b920d19358 fetch-rebase-pull-request: Fix shellcheck warnings.
In tools/fetch-rebase-pull-request line 15:
git checkout -B "review-${request_id}" $remote/master
                                       ^-- SC2086: Double quote to prevent globbing and word splitting.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Anders Kaseorg e7e25fb70b do-destroy-rebuild-test-database: Fix shellcheck warnings.
In tools/do-destroy-rebuild-test-database line 6:
"`dirname "$0"`/../tools/setup/generate-fixtures" --force
 ^-- SC2006: Use $(..) instead of legacy `..`.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-08-03 09:15:26 -07:00
Rhea Parekh ee37866687 import: Add gitter import file in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Rhea Parekh b8e1e8b31d import: Add slack import files in zerver/data_import directory. 2018-08-01 11:52:14 -07:00
Anders Kaseorg 510c97d861 scripts: Use shell quoting when displaying commands to be run.
This way, commands with arguments containing whitespace or
metacharacters are unambiguously readable.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Anders Kaseorg de5e3d3fbd tools/review: Don’t pretend to emulate shell=True either.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:08 -07:00
Anders Kaseorg 9a99d76296 tools: Remove pointless uses of shell=True.
Which, not coincidentally, is all of them.  Don’t use shell=True.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-30 22:39:06 -07:00
Steve Howell 95d136ca5e Extract search_util.js module.
We probably should have done this a while ago, even
though these functions are pretty tiny.  The goal here
is to make it easier to have more consistent search
semantics.

Our first use case is subs.js.  In this case we
are able to decouple a bit of generic string
matching from the subs-specific code.
2018-07-30 11:25:32 -07:00
Greg Price 4ba8015002 tools/reset...: Add experimental option to record the PR as a ref.
I often find myself looking manually through the reflog of `master` to
find a commit I previously reset to with tools/reset-to-pull-request .
Sometimes I want to see a previous version of a PR I'm reviewing a
revised version of; sometimes to look at two related PRs together.

So, here's a feature to automate that by saving each PR branch in its
own ref, with a name like `refs/remotes/pr/1234` -- or `pr/1234`, as
you'd normally refer to it.

To enable this, set the new config option:
  $ git config zulip.prPseudoRemote pr
(Or you can pick another name.)

The reason I hesitate to just make this the behavior for everyone
immediately is that the resulting `pr/1234` refs will naturally
accumulate and may clutter up the view -- and because with the
`refs/remotes/` style of name I've chosen, it requires a bit of
Git plumbing to clean them up.  (Use `git update-ref -d`.)
I'll play with it and iterate; comments welcome from other willing
early adopters.
2018-07-27 23:47:32 -07:00
Harshit Bansal cf5b2b4815 emoji: Change emoticon mapping for `:)`, `(:` and `:(`.
See discussion on CZO:
https://chat.zulip.org/#narrow/stream/101-design/subject/emoji.20picker/near/617811
2018-07-26 11:17:03 -07:00
Anders Kaseorg 19f37fb3d4 tools/test-queue-worker-reload: Avoid shelling out for touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Anders Kaseorg d3f8f74ac4 tools/test-locked-requirements: Avoid shelling out for cp, mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Anders Kaseorg dfed7da911 tools/lib/provision.py: Avoid shelling out for mkdir, touch.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-25 16:54:46 -07:00
Aditya Bansal 0042694e24 stylelint: Start using stylelint to lint our css.
Tweaked by tabbott to document it.

Fixes #8894.
2018-07-25 15:03:35 -07:00
Priyank Patel 8a012b9e7f Revert "webpack: Combine archive-style bundle into archive."
This reverts commit d1d9d63841, and fixes
the fonts in webapp being overwritten by bootstarap due to webpack bundling.
2018-07-24 16:00:26 -07:00
Greg Price 79dfe6e0ae push-to-pull-request: Fix shell usage in an error path.
Errors go to stderr, not stdout.

While we're here, match the message itself to the others in
this script.
2018-07-23 23:19:27 -07:00
Greg Price 26d0904436 push-to-pull-request: Give a nice error if `jq` missing. 2018-07-23 23:19:27 -07:00
Priyank Patel d1d9d63841 webpack: Combine archive-style bundle into archive. 2018-07-23 15:19:47 -07:00
Priyank Patel f8398238a9 webpack: Migrate archive bundle to webpack.
Tested by visiting localhost:9991/archive/streams/1/topics/denmark3.
2018-07-23 15:19:44 -07:00
Harshit Bansal c0b0fb7cce emoji: Move `EMOTICON_CONVERSIONS` mapping to build_emoji infra.
This commit closes a long pending issue which involved moving the
`EMOTICON_CONVERSION` mapping to build_emoji infrastructure so
that there is only one source of truth. This was pending from the
time when this feature was implemented.
2018-07-23 12:35:08 -07:00
Joshua Pan 565fd75661 tests: Test closing alert word status msg. 2018-07-23 14:53:49 -04:00
Tim Abbott 2918cc92f3 search: Duplicate HTML template content for search area.
This is preparation for being able to work on the search pills feature
without making any user-facing changes until we're ready to enable it.
2018-07-23 11:29:10 -07:00
Rhea Parekh 3a6921e54b gitter import: Add gitter data conversion script. 2018-07-23 08:36:30 -07:00
Harshit Bansal bf70955c0c emoji: Update `emoji-datasource` packages.
This commit updates the `emoji-datasource` packages to version 4.0.4.
This update brings following changes to emoji infra:

1: Fix for the bleeding sprite sheets.

2: The category of some emojis has been changed. Categorywise breakup of
   net gain or loss is as follows:
    Travel & Places: 58 (gain)
    Symbols: 47 (loss)
    Smileys & People: 52 (gain)
    Objects: 11 (loss)
    Food & Drink: 3 (gain)
    Animals and Nature: 46 (gain)
    Activities: 9 (loss)

3: There were some changes in the image farm of the package which were
   breaking our old emoji farm. I fixed them by modifying the remapped
   emoji map.

Fixes: #8235.
2018-07-23 08:04:58 -07:00
Harshit Bansal 6c28a60db0 emoji: Hardcode octopus emoji.
Google emojiset's octopus is really cute and whole Zulip community
loves it. So using a CSS hack, we hardcode octopus emoji to use image
from Google's emojiset only irrespective of the choosen emojiset.
2018-07-23 08:04:58 -07:00
Harshit Bansal 3f73826ffa emoji: Extract `get_emoji_code()`. 2018-07-23 08:04:58 -07:00
Harshit Bansal 69eaa2de67 emoji: Move `remapped_emojis` list to emoji_setup_utils.py. 2018-07-23 08:04:58 -07:00
Joshua Pan 015bc8b01e tests: Reach 100% coverage for keydown_util.
This commit pretty much just gets line coverage.
2018-07-23 08:42:11 -04:00
Joshua Pan 0596632457 lint: Exclude frontend_tests/node_tests from $().text rule. 2018-07-20 12:13:56 -07:00
Steve Howell b9c9dd9b5a lint: Turn off the CSS checker (officially).
Our CSS checker globs for .css files.  Since the
SCSS cutover, it has been a no-op, so there's no
sense launching it.  See #8894 for details on
future plans.
2018-07-20 10:03:28 -04:00
Anders Kaseorg 8698198452 tools/update-authors-json: Avoid shelling out for cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 6a8f0165ee tools/test-help-documentation: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg f01942e510 tools/test-emoji-name-scripts: Avoid shelling out for cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 07f5b81406 tools/minify-js: Avoid shelling out for mkdir, cp.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Anders Kaseorg 2d12b5b3d9 tools/diagnose: Avoid shelling out for pwd.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Tim Abbott 811ac718d6 build_emoji: Document a quirk of generate_sprite_css_files. 2018-07-13 21:19:26 +05:30
Harshit Bansal ec1ce08523 emoji_setup_utils: Migrate to python3 type annotations. 2018-07-13 21:19:26 +05:30
Harshit Bansal f6fe5d6266 build_emoji: Migrate to python3 type annotations. 2018-07-13 21:19:25 +05:30
Harshit Bansal f636882e04 build_emoji: Migrate to use `emoji_names.py` file.
This migrates Zulip to use a dramatically better set of names and
aliases for our emoji set, defined in emoji_names.py (which is in turn
manually generated from our hand-curated CSV file).

This should significantly improve the experience of using Zulip's
emoji picker and emoji typeahead for finding what one is looking for.
2018-07-13 21:18:02 +05:30
Tim Abbott 7a86e83a6f build_emoji: Use with for writing to files.
This is a nice code cleanup.
2018-07-13 21:07:08 +05:30
Tim Abbott 85243c525d build_emoji: Pass around emoji_map data structure less.
It wasn't used in either of these functions.
2018-07-13 21:04:37 +05:30
Harshit Bansal b9f974ba61 tools: Add a tool to validate emoji name listing scripts.
This tools emoji name import/export tooling by verifying that no
information gets modified in a round-trip conversion.
2018-07-13 20:53:25 +05:30
Harshit Bansal 628e868d1e tools: Add `export_emoji_names_to_csv`.
This tool is used for exporting `emoji_names.py` to a csv file.
2018-07-13 20:53:08 +05:30
Harshit Bansal 05f85eb94d emoji: Add emoji_names.py file.
Credits to @rishig, Alice Lai, and @rntharu for naming all the emoji.

Names are inspired by iamcal, gemoji, and unicode names, sources like
emojipedia and iemoji, google search results for articles about emoji,
and emoji usage on twitter.
2018-07-13 20:53:07 +05:30
Harshit Bansal ae17a15b29 tools: Add generate_emoji_names_table tool.
This tool generates a listing of the emojis as per the current cache
in use.
2018-07-13 20:53:07 +05:30
Harshit Bansal fef5b43e22 minor: Fix directory name label in 'import_emoji_names_from_csv'. 2018-07-13 20:53:02 +05:30
Max Nussenbaum 8fd900626d billing: Style billing page.
This adds styles to /billing.
2018-07-13 18:30:38 +05:30
Tim Abbott 3cc93fd721 setup_venv: Fix missing libssl-dev dependency.
We were already correctly including libssl-dev in Zulip's dependencies
in development environment provisioning, but (at least now) it's
needed to build certain Python packages like pycurl when building a
Zulip virtualenv in production.  I haven't investigated why we didn't
need this on Ubuntu, but one possible reason would be that some other
library in our dependencies list happens to depend on it on Ubuntu.

We fix this by moving the dependency over to the shared
VENV_DEPENDENCIES list.

Fixes part of #9946.
2018-07-13 18:00:38 +05:30
Harshit Bansal 8ada7cfe5b tools: Add import_emoji_names_from_csv tool. 2018-07-12 18:05:02 +05:30
Greg Price 36cf898589 mypy: Add a place to put type stubs.
This will allow us to begin to add our own stubs for external
libraries.  Writing stubs can be surprisingly little work to do, and
can have high leverage in keeping our type annotations high-quality.
2018-07-12 14:10:09 +05:30
Yago González a3d42d9901 test-api: Pass non-admin client to the test helpers. 2018-07-11 23:54:00 +05:30
Cynthia Lin 7d9281fa6c simplebar: Add simplebar dependency for help page scrolling. 2018-07-11 20:04:55 +05:30
Puneeth Chaganti 3dace27ed6 tools: Only files starting with same migration number conflict. 2018-07-10 21:09:34 +05:30
Shayan Toqraee 0757d022f5 messages: Add support for right-to-left messages.
This implements right-to-left message automatic detection support in
the compose box as well as the message feed.  Full unit tests and
support in the message-editing UI are for future work (as are
potentially more fancy things like supporting things like
right-to-left multi-word names for users/streams/etc.).

Fixes #3123.
2018-07-10 10:47:56 +05:30
Rishi Gupta 16334a1ba7 billing: Update billing system. 2018-07-09 14:33:07 +05:30
Aditya Bansal b6ba162769 generate-fixtures: Extract zulip_test_template creation as a function.
This will be helpful in the upcoming changes which will make use
of this extracted function to re-create zulip_test_template after
migrating zulip_test db so that we have latest schema in tests.
2018-07-09 09:11:48 +05:30
Armaan Ahluwalia ba6543b850 webpack: Fix missing exposed modules left out of transition.
This commit fixes some modules that were erroneously left out while
transitioning app.js to webpack. This commit exposes them using
expose-loader or setting them directly to window.
2018-07-07 08:00:34 +02:00
Shubham Dhama ced6f967bc voting_widget: Increase node test coverage to 100%. 2018-07-06 11:30:12 -04:00
Tim Abbott 0bd139af0c webpack: Fix missing expose for toMarkdown. 2018-07-05 13:24:19 +02:00
Armaan Ahluwalia 54d3d8e8b3 webpack: Transition app.js to be compiled by webpack.
This commit moves all files previously under the 'app' bundle in
the Django pipeline to being compiled by webpack under the 'app'
entry point. In the process, it moves assets under the app entry
to a file called app.js that consumes all relevant css and js files.

This commit also edits the webpack config to be able to expose certain
variables for third party libraries that are currently required by
some modules. This is bad coding form and should be refactored to
requiring whatever dependencies a module may have; we're just
deferring that to the future to simplify the series of transitions we
need to do here. The variable exposure is done using expose-loader in
webpack.

The app/index.html template is edited to override the newly introduced
'commonjs' block in the base template. This is done as a temporary
measure so as not to disrupt other pages on the app during the transition.

It also fixes the value of the 'this' context that was being inferred
as window by third party libraries. This is done using imports-loader
in the webpack config.  This is also messy and probably isn't how we
want things to work long term.
2018-07-05 11:03:08 +02:00
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Shubham Dhama b938523592 widgets: Increase node tests coverage to 100%. 2018-07-03 08:48:49 -04:00
Yashashvi Dave 2b478268f3 third: Add sortablejs library. 2018-07-01 02:05:00 -07:00
Vishnu Ks e1bf749d3c provision: Run build_pygments_data only if required. 2018-06-23 00:15:39 +05:30
Vishnu Ks 109fa85614 provision: Rename file_hash_updated to file_or_package_hash_updated.
Check for changes in package version as well along
with the files.
2018-06-22 23:40:31 +05:30
Vishnu Ks 431d9d6076 provision: Run generate-custom-icon-webfont only if required. 2018-06-22 23:40:31 +05:30
Shubham Padia 6853530eba search: Add search_pill.js with basic operations.
Adds search_pill.js to the static asset pipeline. The items
for search pill contain 2 keys, display_value and search_string.
Adding all the operator information i.e the operator, operand and
negated fields along with the search_string and description was tried out.
It was dropped because it didn't provide any advantage as one had to
always calculate the search_string and the description from the operator.
2018-06-22 09:05:36 -04:00
Shubham Dhama c167227ddc search: Add 100% coverage for search.js. 2018-06-22 07:58:06 -04:00
Greg Price 0462e85d02 tools: Make remote configurable in {reset,push}-to-pull-request.
The appropriate name for the remote pointing at the repo we maintain
may be `upstream` for most of our repos... but not when we're
downstream of someone else, e.g. for react-native.  So, make it easy
to configure per-repo.
2018-06-21 17:28:21 -07:00
Vishnu Ks 59169cc302 provision: Run inline-email-css only if required.
This saves a few hundred milliseconds on provision time.

Edited by tabbott to include email.css in the dependencies.

Fixes #9468.
2018-06-18 06:58:55 -07:00
Vishnu Ks 7b8e79ae48 provision: Refactor hashing of compilemessages into a function.
This allows it to be reused for other tools.

Edited by tabbott to remove the use of "compilemessages" in variable
names.
2018-06-18 06:55:36 -07:00
Raymond Akornor e82581e5bd provision: Close opened files automatically. 2018-06-12 13:38:58 -04:00
Aditya Bansal 1205e02c64 provisioning: Don't rebuild DB if running migrations is sufficient.
This results in a significant optimization in the performance of
re-provisioning Zulip if all that you're doing is rebasing onto a
newer version of master (which just adds new migrations).

The change carries some risk of generating unpleasant-to-debug
situations, because if we merge a buggy migration and then later fix
it, some clients may not have a properly migrated database (and also,
this changes how populate_db commutes with migrations).  But it seems
worth it, given how much time is currently wasted by not having this.

Fixes: #9512.
2018-06-06 13:43:59 -07:00
Aditya Bansal 65dc80fe9d test_fixtures: Add run_generate_fixtures_if_required function.
In this commit we are adding run_generate_fixtures_if_required,
a new function which is meant to de-duplicate a bit of code
between test-server and test-backend which is essentially
responsible for rebuilding the test database if that was required.
2018-06-06 13:37:35 -07:00
Aditya Bansal f7c11d1747 test_fixtures: Refactor to have template_database_status API.
In this commit we are essentially just refactoring the function
is_template_database_current to be called template_database_status
and adjusting the return values accordingly.
This is essentially a preparatory commit for the upcoming commits
which will essentially enable us to not throw away entire DB and
rebuild from scratch if only running migrations could do the job.
2018-06-06 13:37:35 -07:00
Tim Abbott e9312b921c tests: Disable test-queue-worker-reload test.
This test is low value (it's for testing a very specific feature/bug),
is slow, and apparently flakes in CI occasionally.
2018-06-05 09:24:03 -07:00
Tim Abbott 4cdcb7e0d3 webpack: Use more consistent bundle naming scheme.
This way, both our JS and CSS files using .chunkhash.ext as their
naming scheme.
2018-06-03 16:49:59 -07:00
Tim Abbott 7813376934 webpack: Use filenames based on hashes.
This should avoid us creating duplicate webpack bundles every time we
do a deployment, even if none of the files in the bundles themselves
have changed at all.
2018-06-03 16:49:43 -07:00
Tim Abbott 714ad67db1 storage: Move library code into zerver/lib.
There's no particularly good reason for this to have been a top-level
file.
2018-06-03 16:02:02 -07:00
Shubham Padia e32dd53cd9 lint: Check for occurrences of `.includes` except in `frontend_tests/`.
Adds a custom check to js_rules in `/tools/lint/lib/custom_check.py`.
2018-06-03 14:30:22 -07:00
Greg Price f686d55fa0 push-to-pull-request: Broaden one error message.
I discovered this case just now while testing other changes.
2018-06-01 16:52:18 -07:00
Greg Price 954c71e90c push-to-pull-request: Use `jq -r` to simplify a test slightly.
This option (aka `--raw-output`) prints a string as itself, rather
than JSON-encoded; which makes it fit a bit better in a shell script,
saving us a layer of quoting.
2018-06-01 16:52:18 -07:00
Greg Price 8e81ca0fb2 push-to-pull-request: Fix regex usage in parsing remote.
This replaces ad4617c95 with a different fix for the same issue:
instead of stripping the `.git` off separately, we can just correct
the regex, using `+?` to fix our stepping in a classic regex pitfall.
2018-06-01 16:52:18 -07:00
Tim Abbott 751c602a0b update-prod-static: Don't copy/minify templates and styles.
This is a performance optimization: Rather than copying these files
into the `prod-static` directory and then deleting them, we just don't
copy them over in the first place.

For styles, it might have once been the case that this did something,
but we've moved them all to being managed by webpack some time ago.

For the js directory, I think it was never useful to copy and then
delete them; these files were always compiled via tools/minify-js,
and the raw JS files weren't needed, anyway.
2018-06-01 16:39:35 -07:00
Tim Abbott cefdf6f6d4 update-authors-json: Pretty-print output with sorted keys.
This makes working with this file for debugging/etc. much nicer.
2018-06-01 16:39:04 -07:00
Tim Abbott 60eae53327 push-to-pull-request: Provide better output for invalid PR.
Previously, we were very confusingly showing the error message for
permissions when the issue was a not-found error.
2018-06-01 16:00:27 -07:00
Tim Abbott ad4617c953 push_to_pull_request: Fix handling of .git.
This was throwing the "you can't access the repo" error for this
reason.
2018-06-01 16:00:27 -07:00
Tim Abbott 95f1f1d363 mypy: Remove logic for linecoverage report. 2018-06-01 11:14:32 -07:00
Vishnu Ks 69f8b7e8df circleci: Add a base image for Ubuntu Bionic. 2018-05-31 18:51:32 -07:00
Tim Abbott 5b5d1f3e87 update-prod-static: Rewrite to use the run() scripting tool.
This provides nice `set -x` style logging of which commands are
running when.
2018-05-31 16:45:51 -07:00
Sampriti Panda e95f972e24 migrate: Add do_batch_update method for running batch updates.
* Refactor pgroonga_0002 migration to use new method.
2018-05-31 13:00:34 -07:00
Tim Abbott a2c9517f8d test_server: Don't continuously recompile handlebars templates.
This changes run-dev.py to ensure that we have in fact compiled
handlebars templates before running webpack, which is the right model.

Future work will likely include running the handlebars compiler from
webpack, and thus eliminating this extra process.
2018-05-30 20:16:06 -07:00
Tim Abbott 5a35507a7a success-http-headers: Add X-Frame-Options. 2018-05-30 09:41:50 -07:00
Tim Abbott 0fabff6dda setup_venv: Clean up VENV_DEPENDENCIES using parsed lsb_release.
This is mostly a cleanup, but it should also save 50ms in the runtime
of create-production-venv.
2018-05-29 10:57:36 -07:00
Raymond Akornor 5d39a0f0fc scripts: Replace calls to lsb release with our own parsing.
This improves the performance of these operations, by saving a ~50ms
Python process startup.  While not a major performance improvement, it
seems worth it, given how often these commands get run.

Fixes #9571.
2018-05-29 10:57:36 -07:00
Robert Hönig afa806a0ce linter: Add rule to favor Botserver over botserver or bot server. 2018-05-29 07:27:36 -07:00
Tim Abbott 2f571f9a45 webpack: Move font and pygments CSS to common.css.
First, it's silly that these weren't in common.css in the first place,
since that meant these were a bunch of duplicated code, but
additionally, that meant that these weren't available on the
`/activity` page (or other pages that don't include the portico styles).

Fixes #9561.
2018-05-27 23:19:40 -07:00
Yago González b4084d30d2 emails: Move user role logic for followup_day1 to notifications.py. 2018-05-26 07:29:58 -07:00
Yago González 5b0b9feb7d i18n: Atomize translation tags.
Makes the i18n strings in this file much easier to translate by splitting
them into smaller chunks (which avoids having a lot of code in the tagged
strings), and adds a string that was missing as well.
2018-05-26 07:06:43 -07:00
Tim Abbott 20aa3233b0 lint: Improve the check for realm->organization to support identifiers. 2018-05-25 12:03:28 -07:00
Tim Abbott 07b4da7659 Revert "node tests: Change timezone for node tests."
This reverts commit 311ddefd32.

This was causing tests to fail in some but not other environments.
2018-05-25 08:52:13 -07:00
Vishnu Ks abade460a9 tools: Add tool for listing outdated pip packages. 2018-05-24 15:16:02 -07:00
Aditya Bansal f9aa012486 linter: Fix issue with check-templates output.
We fix the issue of check-templates spitting out diff between
expected and found indentation of a file before mentioning the
error message and the file name. Basically stuff was being in the
wrong order despite the fact that in code stuff was happening in the
correct order ie, first print the error message along with the filename
and then the actual diff between expected and found file indentation.

Fixes: #9533.
2018-05-24 12:42:32 -07:00
Vishnu Ks 54a002c2e2 requirements: Upgrade pyflakes to 2.0.0.
We fix a few errors that only the new version finds.
2018-05-24 11:31:36 -07:00
Steve Howell 1b57e568ff Add "zform" functionality to the web client.
A "zform" knows how to render data that follows our
schema for widget messages with form elements like
buttons and choices.

This code won't be triggered until a subsequent
server-side commit takes widget_content from
API callers such as the trivial chat bot and
creates submessages for us.
2018-05-24 09:30:22 -07:00
Steve Howell 435719c65b Add a schema checking concept to the web app.
This starts the concept of a schema checker, similar to
zerver/lib/validator.py on the server.  We can use this
to validate incoming data.  Our server should filter most
of our incoming data, but it's useful to have client-side
checking to defend against things like upgrade
regressions (i.e. what if we change the name of the field
on the server side without updating all client uses).
2018-05-24 09:30:22 -07:00
Steve Howell d018995409 node tests: Make the failure message more obvious.
I mistakenly pushed a PR when my tests failed.  I ran with
the coverage option, so I saw this brightly colored summary
report that distracted me from the failure message.

This adds a couple newlines and some all caps.
2018-05-24 09:30:22 -07:00
Sivakar Sithamparanathan 311ddefd32 node tests: Change timezone for node tests.
The timezone environment variable was set to UTC initially. It was
changed to something other than UTC so that any local vs UTC
conversion issues will manifest in the tests.

Fixes: #5105.
2018-05-23 13:29:19 -07:00
Aditya Bansal 8cfb437677 provision: Don't run create_realm_internal_bots management command.
We essentially stop running create_realm_internal_bots during
every provisioing and move its operations to run from populate db.
In fact to speed things up a bit we actually make populate db call the
funcs which create_realm_internal_bots calls behind the scenes.

Fixes: #9467.
2018-05-23 13:34:52 +05:30
Aditya Bansal a27c9a08dd generate-fixtures: Export DJANGO_SETTINGS_MODULE=zproject.test_settings.
This is required because the --settings=zproject.test_settings param
doesn't work with migrate or the dumpdata management commands. Thus
untill now if one ran just this tool ended up with test database not
properly setup. We never noticed this because test-backend ran this
tool again (after exporting DJANGO_SETTINGS_MODULE) thus making the
tool work this time.
2018-05-23 11:32:40 +05:30
Sampriti Panda 51c71fd3fd provision: Add libfontconfig1 to bionic apt dependencies. 2018-05-22 22:37:04 -07:00
Shubham Dhama ec4e5e73ab org settings: Make time limit input elements inline to dropdown.
(To make the capitalization pass we added `N` to ignored phrases.)
2018-05-22 19:32:44 -07:00
Greg Price 5e89c33e7f tools: Add push-to-pull-request tool.
I've often done this by hand -- basically typed out the last line,
with the variables found from looking at the PR page in a browser.
Seems nicer for both us maintainers and the contributor, in particular
because the PR gets marked as merged, instead of closed.  But it's a
bit of a pain, and I do it maybe half the time or less; plus it's kind
of a subtle GitHub feature, and as a result I think other maintainers
of Zulip repos do this approximately never.

I've always figured this couldn't be hard to automate; today I decided
to take the 45 minutes to look up how, write out the script, QA it,
write up a nice usage message and some comments, and commit it. :)
2018-05-22 18:33:51 -07:00
Armaan Ahluwalia 1525e92058 logging/errors/webpack: Improve error stack traces with blueslip.
This commit improves the output that blueslip produces while
showing error stack traces on the front-end. This is done by
using a library called error-stack-parser to format the stack
traces.

This commit also edits the webpack config to use a different
devtool setting since the previous one did not support sourcemaps
within stack traces. It also removes a plugin that was obviated
by this change.
2018-05-22 14:36:41 -07:00
Eeshan Garg e0ef831993 webhooks: Migrate to UnexpectedWebhookEventType.
This improves test coverage for a lot of our webhooks that relied
on ad-hoc methods to handle unexpected event types.

Note that I have deliberately skipped github_legacy, it isn't
advertised and is officially deprecated.

Also, I have refrained from making further changes to Trello, I
believe further improvements to test coverage should be covered
in separate per-webhook commits/PRs.
2018-05-22 08:30:19 -07:00
Eeshan Garg 3ed20589f2 webhooks: Add generic exception for unexpected webhook events.
UnexpectedWebhookEventType is a generic exception that we may
now raise when we encounter a webhook event that is new or one
that we simply aren't aware of.
2018-05-22 08:30:19 -07:00
Michael J. Sullivan c81b276c1c mypy: Add basic support for invoking dmypy to run-mypy. 2018-05-21 22:41:08 -07:00
Aditya Bansal e41d32a15a reminders: Add various design improvements for the UI/reminder msg. 2018-05-21 09:07:37 -07:00
Tim Abbott c6fda61227 block_internet: Make compatible with httpretty library.
If httpretty is enabled, we can let it handle HTTP requests.
2018-05-20 23:31:42 -07:00
Tim Abbott a170bd70af provision: Don't block installing with Ubuntu bionic.
Now that we have tsearch_extras packages uploaded, this mostly works.

There's a few issues being debugged in #9460; they should be fixed
soon, and regardless, merging this will make it easier to develop.
2018-05-20 21:21:11 -07:00
Tim Abbott 06ed55e45a webpack: Add ALLOWED_HOSTS list for zulipdev.com and friends.
This makes it possible to again use the *.zulipdev.com domains in the
development environment.

Ideally, we'd also read REALM_HOSTS to make this more flexible.
2018-05-20 18:12:28 -07:00
Max Nussenbaum 0ff2051982 portico: Add tour section to homepage.
This adds a tour of Zulip to the bottom of the homepage.

In order to get the carousel nave, we use Bootstrap 2 from a CDN on
this page; this isn't ideal in the medium term, but upgrading
Bootstrap across the project is too much work for now.
2018-05-20 15:04:23 -07:00
Tim Abbott 4f8e09d5af provision: Fix buggy management of apt_dependencies_hash.
Apparently, we were incorrectly appending each new hash onto the end
of the file, basically resulting in every run of provision being
treated as a miss for this cache.

Fixing this saves about 4s (over 1/3) of the no-op provision time.
2018-05-20 14:22:32 -07:00
Tim Abbott 235002a549 provision: Don't install lsb-release if already installed.
This early bootstrapping step should be rearely required, and it saves
about 0.5s in our no-op provision time.
2018-05-20 14:22:32 -07:00
Shubham Padia 0824308a7a tools: Allow optional arguments after file arguments in test_backend.
Fixes #9233.
Uses nargs='*' instead of nargs='argparse.REMAINDER'.
nargs='argparse.REMAINDER' gathers remaining terms as arguments
even if it is an option e.g --coverage, while '*' gathers all the
command-line arguments until the next option is encountered.
2018-05-18 17:08:40 +05:30
Tim Abbott 0dcf873370 third: Rename thirdparty-fonts.css for better clarity.
Now it's clear that this is just legacy fontawesome styles.
2018-05-17 04:58:30 -07:00
Aman Ghebreweldi b05a702285 third: Use font awesome css file from NPM package. 2018-05-17 04:58:30 -07:00
Steve Howell b48f052b0c Add basic widgets framework (JS side). 2018-05-16 15:13:33 -07:00
Steve Howell 1f0ffdc415 Add widget.py to enable widgets on the server side.
The only slash command implemented in this initial
version is an extremely crippled version of a
"/stats" slash command that reports that you are
running 1 server.
2018-05-16 15:13:33 -07:00
Joshua Pan a67ba617d6 minor: Fix typo in tools/lint. 2018-05-16 13:19:28 -04:00
Tim Abbott 00861b910d test_presence: Add tests for seconds_usage_between.
This isn't widely used, but is a nice way to verify the logic further.
2018-05-15 18:37:49 -07:00
Tim Abbott e39a3e07d7 coverage: Mark zerver/lib/debug.py as not needing coverage. 2018-05-15 17:19:14 -07:00
Tim Abbott fd24eac8bb coverage: Exclude zerver/lib/parallel.py from coverage. 2018-05-15 17:19:14 -07:00
Tim Abbott c46149efcb migrate: Remove obsolete act_on_message_ranges.
This has a cool structure, but it's written against the long-dead
South API, and we can always pull it out of the Git history if we want
to use this approach in the future.
2018-05-15 17:19:14 -07:00
Tim Abbott 3e04aa99fa fonts: Use source-sans-pro CSS from upstream. 2018-05-15 16:22:44 -07:00
Greg Price 6dbd90835b mypy: Move remaining exclusion list to config file.
This leaves the wrapper script with very little left to do!

The main thing left is finding scripts by searching for shebang lines;
mypy itself would happily do the search for importable Python files.
2018-05-15 18:14:33 -04:00
Tim Abbott badfec2b49 coverage: Don't include settings files in test coverage. 2018-05-15 13:55:01 -07:00
Tim Abbott 191e6dc13a create_user: Remove user_profile_id option.
This hasn't been used in years, had no test coverage, and doesn't have
a clear use case.
2018-05-15 13:51:56 -07:00
RobbieClarken f81b936727 zerver/tests: Require 100% test coverage of zerver/lib/upload.py.
The last line here is impossible to test.

Fixes #4489.
2018-05-15 10:52:20 -07:00
Aman Ghebreweldi 83ee8211a8 third: Get sorttable package from npm.
We didn't have any local changes, so there's no need to have it
vendored in the repository.
2018-05-15 09:25:48 -07:00
Steve Howell 4f0b3fc1b6 test-js-with-node: Exit more cleanly for errors.
We don't need a Python stack trace when the node
piece fails, so just give a non-distracting error
message.
2018-05-15 08:24:44 -07:00
Joshua Pan df84e1d7eb tests: Reach 100% coverage for zerver/liv/management.py.
Cleaned up add_user_list_args(). The "help" and
"all_users_help" have all default values. As noted in
an earlier commit, "all_users_help" is always passed in,
so we can get rid of "all_users_arg". We keep the default
for "all_users_help" so we don't have to change variable order
in function definition.
2018-05-14 10:46:21 -07:00
Darshan Markandaiah 607cab2a53 tests: Add 100% test coverage to zerver/apps.py.
Tweaked by tabbott to rename to test_cache.py and remove the
sender_name argument (Since it was kinda confusing).
2018-05-14 10:36:10 -07:00
Vishnu Ks b73603a97c droplets: Do shutdown command along with clear history.
Otherwise shutdown command would be still present in the
history.
2018-05-14 08:12:43 -07:00
Vishnu Ks 861001f7b3 droplets: Update snapshot id.
Upgraded zulip and python-zulip-api repos.
Installed flask in python-zulip-api.
2018-05-14 08:12:43 -07:00
Tim Abbott f5ea661c1f tools: Upgrade pycodestyle.
We have to add a few rules that we're ignoring for now.
2018-05-14 06:15:13 -07:00
Aditya Bansal 64678b459c linter: Add rule to lint against use of typing.Text and promote str.
This commit is the last in a series of commits which migrated our
entire codebase to use str instead of typing.Text.

Fixes: #9203.
2018-05-13 17:17:32 -07:00
Aditya Bansal addfde7374 tools: Change use of typing.Text to str. 2018-05-13 17:17:32 -07:00
Joshua Pan 073ecaac66 provision: Give concrete NFS error message on older OSX versions. 2018-05-13 08:47:19 -07:00
Tim Abbott ff2157c787 coverage: Exclude api_test_helpers.py from backend test coverage.
These files are tested by a different test suite, and in practice have
100% coverage; no need to double-measure it.
2018-05-09 20:49:13 -07:00
Tim Abbott 7e2841b358 bulk_create: Mark bulk_create_streams as nocoverage.
It's only used in `populate_db`, so it gets tested that way.
2018-05-09 20:49:12 -07:00
Steve Howell 27b7461e0a Revert "js: Implement DynamicText class."
This reverts commit 6e7305f784.

We never ending up using this class.
2018-05-08 12:13:55 -07:00
Vishnu Ks 49b3cb9da5 droplets: Remove the step to activate virtualenv manually.
I didn't had to activate this manually during the
last few droplet upgrades.
2018-05-07 08:42:18 -07:00
Vishnu Ks c684333051 droplets: Fix the command for clearing bash history.
history -c only clears the history from memory and
would be populated back during the next SSH session.
2018-05-07 08:42:18 -07:00
Vishnu Ks 18314e57f8 droplets: Add origin remote to python-zulip-api as well. 2018-05-07 08:42:18 -07:00
Vishnu Ks 13f6cbeefd droplets: Change droplet region from SFO1 to NYC3.
Since the region of base.zulipdev.org is
NYC3 we had to add SFO1 as an additional
region each time a snapshot of base droplet
is created. This is required as droplets
can be created in SFO1 only if there is
an image present in that region. Adding
of droplet image to SFO1 takes a lot of
time as well as cost 2X as we are storing
2 images. It's better to just create new
droplets in NYC3 instead. Alternatively we
can create a new base droplet in SFO1 if
we want all the droplets to be created in
SFO1.
2018-05-07 08:42:18 -07:00
Vishnu Ks d6c5635550 droplets: Update snapshot id. 2018-05-07 08:42:18 -07:00
Greg Price 9f3052b0ef mypy: Move remaining options on type-check semantics to config file.
This puts all of this config in one place, and also needs a lot fewer
lines to describe it; which, combined, makes it a lot clearer what our
normal config actually is.  (I'd been looking at this script for a few
minutes without realizing that we have `--disallow-untyped-defs` *on*
by default, not off.)

Experimenting with different values is still easy; just comment the
line in the config.
2018-05-06 19:33:55 -07:00
Greg Price 6b1e76c1b1 mypy: Move cache-dir to config file. 2018-05-06 19:33:55 -07:00
Greg Price 724e849e2b mypy: Move follow-imports to config file.
It's cleaner to have this configuration in a nice declarative
config file than embedded in a script.
2018-05-06 19:33:55 -07:00
Greg Price ff178bb27a mypy: Drop now-redundant `-i` option.
This used to be a synonym for `--incremental`.  Since mypy 0.590,
incremental mode is the default, and the flag is ignored; so we
can happily drop it.
2018-05-06 19:33:55 -07:00
Tim Abbott 41841221ee scripts: Remove obsolete zesty configuration.
Zesty already reached end-of-life, so we'll never support it.

And in one place, we add support for bionic.
2018-05-05 11:41:57 -07:00
Tim Abbott d178c53a10 provision: Add official support for provisioning directly on Debian. 2018-05-05 11:00:47 -07:00
Tim Abbott 8ea8bfe285 puppet: Add basic configuration for Ubuntu bionic. 2018-05-05 10:49:09 -07:00
Tim Abbott b193330474 provision: Add tsearch_extras package for stretch. 2018-05-05 10:49:09 -07:00
Tim Abbott 78f3cff151 provision: Fix unescaped reference to vagrant.
Previously, this would actually try to run `vagrant ssh`, rather than
printing it :(.
2018-05-05 10:48:37 -07:00
Tim Abbott 9d6233a457 capitalization: Add an exclude rule for zulip_org_id. 2018-05-03 22:37:24 -07:00
Armaan Ahluwalia cca10beb78 css/webpack: Moved archive styles to webpack bundle.
This commit moves the stylesheets under the archive bundle in
the Django pipeline to being compiled by webpack instead. It
also removes a remaining call to a portico stylesheet that no
longer exists.
2018-05-03 19:15:49 -07:00
Armaan Ahluwalia 54bf2a6231 css/webpack: Transition landing-page.css to webpack.
This commit transitions landing-page.css from the Django pipeline
to being compiled by webpack as landing-page.scss under the
'landing-page' and 'integration' bundles.
2018-05-03 19:15:23 -07:00
Armaan Ahluwalia 1c016e990d css/webpack: Transition common.css to webpack.
This commit transitions common.css from the Django pipeline
to being compiled by webpack under the common bundle.
2018-05-03 19:15:23 -07:00
Tim Abbott bd110ccb3c test-migrations: Add django_two_factor migrations to exclude list.
We don't control the names for these migrations.
2018-05-03 12:23:25 -07:00
Priyank Patel 1e1b72f6c8 webpack: Transition stats.css to use webpack.
The stats.scss file is added to activity bundle.
2018-05-03 08:08:23 -07:00
Priyank Patel c780bc33ba webpack: Transition activity.css to use webpack.
The activity.scss file is added to activity bundle.
2018-05-03 07:53:24 -07:00
Armaan Ahluwalia 5f7b47e20c css: Transition 'app.css' to SCSS.
This commit transitions all styles in app.css in the Django pipeline
to being compiled by webpack in an app-styles bundle, and renames the
various files to now be processed as SCSS.

To implement this transition, we move the old CSS file refernces in
settings.py and replace them with a bundle declared in
`webpack.assets.json` and includedn in the index.html template

Tweaked by tabbott to keep the list of files in `app.css` in
`webpack.assets.json`, and to preserve the ordering from the old
`settings.py`.
2018-05-02 17:13:16 -07:00
Armaan Ahluwalia 64dadae697 webpack: Add css-hot-loader to remove flash on unstyled content.
This commit removes the flash on unstyled content while in dev
mode that was caused by the use of style-loader. Instead it
enables mini-css-extract-plugin in dev in combination with
css-hot-loader which enables HMR for development.

This is because mini-css-extract-plugin does not currently support
HMR out of the box. It also adds a SourceMapDevtoolPlugin to enable
sourcemaps with css since mini-css breaks sourcemaps when used in
combination with the cheap-module-evel-source-map setting.

Related issues:
https://github.com/webpack-contrib/mini-css-extract-plugin/issues/34
https://github.com/webpack-contrib/mini-css-extract-plugin/issues/29
2018-05-02 16:55:58 -07:00
Aditya Bansal a62efd55df linter: Make duplicate html tag id detection work with archives.
We modify check-templates to check for duplicate id's in archive
templates and app templates separately. This means we are allowing
app and archive templates to potentially use same id's. This is
needed because we intend to re use some js from the main app and
having same id's help achieve that.
Note: We haven't up until this point actually added archive
templates. This commit is more of a preparatory commit for merging
the basic archive infra.
2018-05-02 15:23:33 -07:00
Armaan Ahluwalia 9f80418d12 webpack: Add support for png file imports in file-loader.
Adds support for importing png files using file-loader in webpack.
Changes the name of the output directory to be files instead of
fonts for better readability.
2018-05-02 09:45:04 -07:00
Armaan Ahluwalia fce6882eb9 webpack: Move the styles consumed by 5xx.html to webpack.
This commit removes the need for portico.css to be generated
by the Django pipeline and makes the error page use the css
file compiled by webpack instead.
2018-05-02 09:45:01 -07:00
Tim Abbott a4ff917789 emails: Fix spelling of "Unknown IP" and tag for translation. 2018-04-30 12:04:39 -07:00
Priyank Patel 50b13219a3 webpack: Combine both js and css into one portico bundle.
Combines, both portico js and css into one bundle. This for now solve
the issue of an empty js bundle being generated by webpack for the
portico-styles stylesheet.
2018-04-30 10:23:39 -07:00
Tim Abbott 7d6bb3dcb4 settings: Remove obsolete default_desktop_notifications setting.
This actually hasn't been hooked up to do anything in years.

While we're at it, we remove the entire "Zulip Labs" settings page.
2018-04-28 13:46:07 -07:00
Steve Howell 605a90ce36 node tests: Reach 100% coverage for scroll_util.js. 2018-04-28 11:15:14 -07:00
Steve Howell 65d8eb3189 buddy list: Extract user_search.js.
This was a bit more than moving code.  I extracted the
following things:

        $widget (and three helper methods)
        $input
        text()
        empty()
        expand_column
        close_widget
        activity.clear_highlight

There was a minor bug before this commit, where we were inconsistent
about trimming spaces.  The introduction of text() and empty() should
prevent bugs where users type the space bar into search.
2018-04-28 11:15:14 -07:00
Armaan Ahluwalia fb0a421b8c webpack: Silence most webpack output for tests.
This commit adds a --quiet argument to tools/webpack which removes
the verbose output from webpack and replaces it with showing only
errors. It also makes tools/run-dev --tests use this argument while
running webpack for testing.

Tweaked by tabbott to clean up the code a bit.
2018-04-28 09:32:10 -07:00
Tim Abbott 699c4381f2 test-api: Add a PROVISION_VERSION --force option.
This brings this in line with the Casper and other tests.
2018-04-27 17:21:12 -07:00
Vishnu Ks e3314be114 emails: Add translation tags to confirm_new_email. 2018-04-27 11:59:36 -07:00
Armaan Ahluwalia 93ac40105f CSS: Move portico styles to webpack compilation.
static/styles/scss/portico.scss is now compiled by webpack
and supports SCSS syntax.

Changed the server-side templates to render the portico-styles
bundle instead of directly requiring the portico stylesheet. This
allows webpack to handle stylesheet compilation and minification.

We use the mini-css-extract-plugin to extract out css from the
includes in webpack and let webpacks production mode handle
minification. Currently we're not able to use it for dev mode
because it does not support HMR so we use style-loader instead.
Once the plugin supports HMR we can go on to use it for both
dev and prod.

The downside of this is that when reloading pages in the development
environment, there's an annoying flash of unstyled content :(.

It is now possible to make a change in any of the styles included
by static/styles/scss/portico.scss and see the code reload live
in the browser. This is because style-loader which we currently
use has the module.accept code built-in.
2018-04-27 09:04:50 -07:00
Armaan Ahluwalia f20671a509 webpack: Fix Hot Module Reloading in webpack.
This commit fixes hot module replacement in webpack. To do this
we open port 9994 used by webpack to communicate between browser
and devserver. The attempts to forward the proxy from 9991 failed
so the last resort was to open up the webpack port.
It also removes an uncessary plugin in the webpack config and moves
the --hot flag to tools/webpack.
2018-04-27 09:04:49 -07:00
Steve Howell 9ece6d2be4 lint: Exempt "opts." from addClass checks.
If we use something like "opts.highlight_class", it's probably
in a generic widget.
2018-04-26 08:42:47 -07:00
Tim Abbott 1ff909d971 coverage: Exclude Zephyr ccache logic from test coverage.
That code path is pretty constrained in how it's used, and is only for
the legacy Zephyr integration which we don't expect to spend effort
on again.
2018-04-25 22:41:31 -07:00
Tim Abbott 47bdf5ecba coverage: Exclude generate_test_data from testing.
That file is just used for generating manual testing data; it is
actually tested by our suite, just before coverage starts running.
2018-04-25 22:38:54 -07:00
Tim Abbott d5946de718 decorator: Add nocoverage comments for rate_limit decorator.
We've already got a bunch of other comments on work we need to do for
this decorator and an open issue that will ensure we at some point
rework this and add tests for it.  In the meantime, I'd like to lock
down the rest of decorator.py at 100% coverage.

Fixes #1000.
2018-04-25 22:37:12 -07:00
Priyank Patel bc454bab88 webpack: Disable host check for webpack-dev-server.
Webpack dev server by default does host checking for requests. so
in dev enviorment if the the request came for zulipdev.com it would not
send js files which caused dev envoirment to not work.
2018-04-24 14:14:20 -07:00
Emilio Schadt b6f5ea0fd2 docs: Add README.md to zulip/docs to prevent reading on GitHub.
This should help avoid confusing about broken links.

Tweaked by tabbott to fix formatting and the linter.
2018-04-24 10:50:39 -07:00
Vishnu Ks 7533796ea9 popover: Add an option to show user profile.
Fixes #8880
2018-04-23 16:56:24 -07:00
Tim Abbott 1703e23980 templates: Move all core app templates into a subdirectory.
This should make it easier to find the templates that are actually
part of the core webapp, instead of having them all mixed together
with the portico pages.
2018-04-23 16:46:37 -07:00
Tim Abbott 8fc04a074d test-backend: Update coverage excludes for new import_realm.py. 2018-04-23 16:26:01 -07:00
Armaan Ahluwalia 97ec5e0aaa webpack: Tweak config file for stats and faster recomp on HMR.
Update the config file to show slightly more information while
compiling webpack. Also decreased the time webpack waits before
recompiling in order to speed up HMR.
2018-04-23 15:49:33 -07:00
Armaan Ahluwalia c70d26224d webpack: Change devtool sourcemap strategy.
Changed the devtoool setting for development from 'eval' to
'cheap-module-eval-source-map' as it has better support for
breakpoints in Google Chrome and the difference is time is
negligible at the number and size of files currently being
consumed by webpack. This stragtegy can be reviewed in the
future as the size of files grows or Chrome adds better
support.
2018-04-23 15:49:33 -07:00
Armaan Ahluwalia fc7aa1a771 webpack: Upgrade webpack version 4.5.0.
Upgrade webpack to latest version at the time of authoring. This
involves upgrading webpack version and its loaders to compatible
versions. It also involved editing tools/webpack to use the
executable for webpack-cli instead because of a change in how the
webpack package wants you to handle shell execution.
It also fixes the confugration for TypeScript in the webpack config
as that was previously broken. Including TypeScript files in JS
files compiled by webpack now works.
2018-04-23 15:49:33 -07:00
Priyank Patel 29f04511c0 node_tests: Use nyc instead of deceprated istanbul package. 2018-04-23 15:27:16 -04:00
Tim Abbott 3f4f94d111 check-capitalization: Add "G Suite" to proper nouns list.
This fixes an error caught by CI.
2018-04-23 10:21:10 -07:00
Steve Howell 6d4855bd6a lint: Prevent accidental use of const.
This prevents us from using const in our JS code, with exceptions
for test code and the portico.  Hopefully this is just a temporary
rule until we make our pipelines with work with ES6.

I tried to prevent "let", but that was too noisy.

This adjusts the one false-negative case of using const in a comment.
2018-04-22 20:11:38 -07:00
Steve Howell 3f1930f9c5 buddy list: Extract buddy_data.js. 2018-04-22 20:08:08 -07:00
Steve Howell 536236d9b1 buddy list: Extract buddy_list.js. 2018-04-22 20:08:08 -07:00
Eeshan Garg ca5ea20ab7 doc tests: Add portico pages to tools/test-help-documentation.
Fixes #9117.
2018-04-21 21:51:32 -07:00
Preston Hansen 76d6c71595 tests: Move zerver/fixtures to zerver/tests/fixtures for clarity.
Fixes #9153.
2018-04-19 21:50:17 -07:00
Joshua Pan 99f07fe2e2 tests: Fully cover top_left_corner.js. 2018-04-19 14:56:55 -04:00
Eeshan Garg 19177a4aff webhooks: Move github_webhook/ to github/. 2018-04-19 11:00:55 -07:00
Eeshan Garg 48b8558c02 webhooks: Move github/ to github_legacy/ and remove docs. 2018-04-19 11:00:55 -07:00
Steve Howell f8fd169a7d node tests: Allow --coverage for individual files.
We now allow you to run --coverage on individual files.  This helps
when you want to make sure a file is being covered directly and not
just getting incidental coverage from higher level tests.

Before this commit, we were conflating wanting coverage reports with
wanting coverage checks.  For individual files, we now solve that by
simply eliminating the coverage checks.  This required some minor
refactoring to extract some functions.
2018-04-17 17:52:19 -07:00
Vishnu Ks cc93ac34a8 coverage: Add coverage to estimate_recent_messages.
With this message.py is fully covered and can be
removed from not_yet_fully_covered in test-backend.
2018-04-17 11:01:20 -07:00
Alyssa Wagenmaker d4e5777296 tests: Test user unsubscribing before soft deactivation.
Brings lib/soft_deactivation.py up to 100% test coverage.

Improves: #7089.
2018-04-16 11:28:08 -07:00
Tarun Kumar 0b62410f5e pm_pill: Achieve 100% node-test coverage for compose_pm_pill.js. 2018-04-13 11:59:57 -07:00
Tarun Kumar 51601fab44 muting: Achieve 100% node-test coverage for settings_muting.js. 2018-04-13 11:59:57 -07:00
Tim Abbott 105eed049e install-node: Fix leaking of $HOME.
This fixes a bug where provision was failing since our most recent
upgrade to yarn/nvm/node.

It turns out my original fix was the correct fix, but to the wrong
third-party tool: nvm, not yarn, was the offender.
2018-04-12 14:32:36 -07:00
Tim Abbott feef35bf25 linter: Add checks for sloppy use of .html().
Since jQuery's .html() can be a source of security bugs, we add a new
lint rule that tries to catch common problematic uses.
2018-04-12 09:50:14 -07:00
Shubham Padia 55619cbe70 browser-support: Add string.prototype.endswith polyfill.
String.prototype.endsWith is not supported in ie11.
Adds string.prototype.endswith package to dependencies and places
it at `common` entry point in webpack.assets.json.
2018-04-11 15:40:57 -07:00
Shubham Padia 57a494f94d browser-support: Add codepointat polyfill to `common` entry point.
Adds string.prototype.codepointat which was added as a polyfill
earlier to the project but was not added to `common` entry point.
2018-04-09 12:16:19 -07:00
Shubham Padia b906562f22 browser-support: Add string.prototype.startsWith polyfill.
Fixes #8944.
Adds string.prototype.startswith package to dependencies and places
it at `common` entry point in webpack.assets.json. As common.js is
loaded on all code paths first, there is no need to place this package
into other entry points.
2018-04-09 12:16:19 -07:00
Aditya Bansal b9f1acb300 linter: Enforce 2 space indents on tags spread over multiple lines.
We make some specific cases of tags use 2 space indents.
The case description:
* A tag with opening tag spread over multiple lines and closing tag
on the same line as of the closing angle bracket of the opening tag.
* A tag with opening tag spread over multiple lines and closing tag
not on the same line as of the closing angle bracket of the opening
tag.

Example:
Case 1:

Not linted:
<button type="button"
class="btn btn-primary btn-small">{{t "Yes" }}</button>

After linting:
<button type="button"
  class="btn btn-primary btn-small">{{t "Yes" }}</button>

Case 2:

Before linting:
<div class = "foo"
     id = "bar"
     role = "whatever">
     {{ bla }}
</div>

After linting:
<div class = "foo"
  id = "bar"
  role = "whatever">
    {{ bla }}
</div>
2018-04-07 20:08:44 -07:00
Aditya Bansal 550222dede linter: Make multiline handlebar singleton tags use 2 space indentation. 2018-04-07 20:08:38 -07:00
Aditya Bansal 2fe012ffff linter: Make html singleton tags use 2 space indentation. 2018-04-07 20:08:31 -07:00
Alyssa Wagenmaker df666c3dfc test-backend: Report fully covered files still in not_yet_fully_covered. 2018-04-06 17:29:18 -04:00
Ben Reeves 29a079ebbf test-backend: Remove fully covered files from not_yet_fully_covered. 2018-04-06 17:29:18 -04:00
Tim Abbott b0b134cb4c help: Clean up settings_html and subscriptions_html.
After some thinking, I don't think there's any actual value to doing
the ../ style relative links here, whereas there is actual harm from
the links being slightly broken in the current model.  We fix this by
just using /#settings as the URL.

Fixes #8978.
2018-04-05 14:48:26 -07:00
Tim Abbott ed5b374ffa help docs: Fix a sorta buggy URL test. 2018-04-05 13:07:34 -07:00
Steve Howell e145258fee node tests: Add 100% coverage for components.js. 2018-03-29 08:09:38 -07:00
neiljp (Neil Pilgrim) 090b47ed19 mypy: Add explicit Optional for default=None parameters in various files. 2018-03-28 12:31:51 -07:00
neiljp (Neil Pilgrim) 07971d3a66 linter: Add check that Optional models are tagged with null=True.
Exclude NullBooleanField explicitly.

Amend one line in models.py found from this linter change.
2018-03-27 10:21:25 -07:00
neiljp (Neil Pilgrim) 7d93f5726c linter: Add check that null=True lines are annotated as Optional.
Also add two missing annotations of this type that were missed
until linted.
2018-03-27 10:21:25 -07:00
Joshua Pan 0f58f20ad8 tests: Clean up test-js-with-node output on success. 2018-03-26 06:35:58 -04:00
Joshua Pan ed9eb3bdb5 tests: Make test-js-with-node handling smarter.
Added support for passing a filename without `.js` suffix.
This then fixed the issue of no complaints for invalid test
files. Now, throws an error for invalid test files.

Fixes #8579.
2018-03-26 06:35:58 -04:00
neiljp (Neil Pilgrim) 9a49812cde mypy: Extract default run-mypy options into mypy.ini. 2018-03-23 16:53:43 -07:00
neiljp (Neil Pilgrim) fb4f5c8570 tools: remove strict-optional command from run-mypy.
This is now controlled globally and on a per-file basis in mypy.ini.
2018-03-23 16:53:43 -07:00
neiljp (Neil Pilgrim) 70117f0e83 mypy: Add asserts of intermediate values in update-authors-json. 2018-03-23 13:07:39 -07:00
neiljp (Neil Pilgrim) f0e3fcb937 mypy: Amend test-backend:internet_guard to return NoReturn.
It is guaranteed to raise an exception, so this is reasonable,
and it satisfies mypy.
2018-03-23 11:39:54 -07:00
sinwar 675e844e6b team page: Add contributors of zulip-terminal.
It adds zulip terminal contributions to
team page using github apis.
2018-03-22 21:06:18 -07:00
Tim Abbott 08a6271b54 update-authors-json: Improve error handling for duplicate commits.
It's possible that a user with duplicate commits might not be in the
top 100 users for a given repo.
2018-03-22 14:58:13 -07:00
Gooca 2363c9955a ui scrollbar: Update perfect-scrollbar to 1.3.0.
Update perfect-scrollbar to fix stutter space-scrolling in #8544. Also
reworked deprecated `element.perfectScrollbar` to `new
PerfectScrollbar(element)`.  Lastly, updated provision version and
changed node module path to new path.

This also refactors perfect-scrollbar in help.js to work with updated
version of perfect-scrollbar. Because the update also changed
perfect-scrollbar's css selectors for all scrollbars in zulip, we
update those too.

Fixes #8544.
2018-03-20 21:59:11 -07:00
Tim Abbott 791953963b check-capitalization: Include banned words in overall output.
This locks down the ban on use of "Realm" in translated strings in the
codebase.
2018-03-16 16:59:02 -07:00
Tim Abbott 8e334b0f64 check-capitalization: Exclude certain uses of the word "realm".
Uses that are actually a variable name don't make sense to exclude.
2018-03-16 16:58:49 -07:00
neiljp (Neil Pilgrim) 966ca7015f mypy: Finalize migration of cache.py to python3 function annotation.
- Use forward declarations of some types from models.py to avoid cycles.
- Remove cache.py from linter rule exclude list to ensure it stays that way.
2018-03-16 11:29:12 -07:00
Vishnu Ks c385fcefa8 templates: Fix translation tag for disposable email tooltip. 2018-03-15 14:35:24 -07:00
neiljp (Neil Pilgrim) a3d8cc1f86 Mypy: Migrate tornado/descriptors.py to python3 function annotations.
Use quoted form of ClientDescriptor to break presumed circular dependency,
and modify import statement to shorten quotes.
2018-03-15 12:54:43 -07:00
neiljp (Neil Pilgrim) a7bfb09067 Mypy: Use models.py QuerySet annotation approach in stream_subscription.py.
Namely, annotate as best as possible, and add notes to indicate preference,
if QuerySet develops generic typing.

Note that the return values of functions with annotations changed in this
commit are used elsewhere as QuerySets, so the Sequence[T] approach used
for some functions in models.py is not applicable.
2018-03-15 12:54:43 -07:00
neiljp (Neil Pilgrim) ce4ac0d2cf Mypy: Ensure consistency of QuerySet return types in models.py.
Other functions took the form of returning Sequence[T] when the QuerySet
functionality is unused beyond the function, with T being the objects
filtered for in the function body; this commit follows that practice for the
one remaining python2 comment-annotated function, completing the transition
of models.py to py3.5 function annotations.

A note is also added to another function regarding a need to return a
QuerySet, and ideally a QuerySet[T] in line with the other functions, as and
when QuerySet becomes annotated as a generic.
2018-03-15 12:54:43 -07:00
Tarun Kumar c8d5cf7d6c user-groups: Achieve 100% node-test coverage of user_groups.js.
Fixes #8704.
2018-03-15 10:09:28 -07:00
Harshit Bansal 40958e0824 emoji: Switch to 64px 256 color indexed sprite sheets.
This commit switches our emoji infrastructure to use 256 color indexed
64px spritesheets. Earlier we were using non-indexed 32px spritesheets
which were blurry on high dpi displays. These indexed spritesheets not
only provide a crispier display but are also smaller in size.

This commit also removes the `emoji-datasource` package as a dependency
as all the data is now sourced from individual datasource packages.

Fixes: #7862.
2018-03-14 10:28:45 -07:00
neiljp (Neil Pilgrim) 43ff4ea1f7 tools: Ensure mypy is run on .pyi files. 2018-03-13 13:49:39 -07:00
Tim Abbott 8866c23b5e test-backend: Enforce 100% test coverage on models.py.
Following the exclusions and code deletions we just did, we can now
lock down 100% test coverage on zerver/models.py.
2018-03-12 11:52:39 -07:00
neiljp (Neil Pilgrim) ee2b8bfbfb linter: Require Py3 function annotations, with some exceptions.
Main exceptions are scripts/tools/puppet & tests.

Other current exclusions:
- api_test_helpers.py (avoid changing test code);
- cache.py, due to comments in file;
- models.py, due to failure on QuerySet[Message];
- stream_subscription.py, due to failure on QuerySet[Subscription];
- tornado/descriptors.py;
- views/streams.py, due to remaining FuncKwargPair issue;
- zthumbor/, since thumbor is in python2.

Tweaked by tabbott to partially document the stauts in comments.
2018-03-12 11:23:30 -07:00
Umair Khan 728cd04c4d check-capitalization: Check for banned words.
Fixes #8619
2018-03-11 15:34:47 -07:00
Steve Howell fef2840531 Add FetchStatus class.
We will use this in subsequent commits to replace the
`load_more_enabled` flag.
2018-03-09 14:20:58 -08:00
sinwar 6bed6ccdcf team page: Remove duplicate contribution counts.
It removes duplicate contribution count from zulip-server.

Fixes #7836.
2018-03-07 17:25:41 -08:00
Steve Howell 3e76d523c7 node tests: Add tests for user_pill.js.
This gets coverage to 100%.  (We already had a lot of indirect
coverage, but these tests should have more obvious failure
symptoms.)
2018-03-07 15:53:11 -08:00