Commit Graph

3704 Commits

Author SHA1 Message Date
Anders Kaseorg 43996106d8 requirements: Get transifex-client from apt.
In the past it has blocked Python library security updates with overly
strict version bounds, and we don’t use it as a library, only as a
binary.

Skip the PROVISION_VERSION bump because we can use the tx binary from
either location.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:39:04 -07:00
Anders Kaseorg 006a69c912 ci: Disable test-run-dev.
This test doesn’t catch many bugs and is known to be flaky.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:38:19 -07:00
Anders Kaseorg a6624f04db Revert "run-dev: Run process_queue with DJANGO_AUTORELOAD_ENV."
This reverts commit 36a8e61e67 (#13934).

The Django 2.2 autoreloader works by forking into a child process that
exits with status 3 when a file changes, and a parent process that
restarts the child when it exits with status 3.  Setting this
environment variable had the effect of pretending we were already the
child process, without a parent process to restart it.  Therefore,
changing any code used by the queue processor caused it to exit rather
than restart.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:38:19 -07:00
Anders Kaseorg 7ff9b22500 docs: Convert many http URLs to https.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:35:32 -07:00
arpit551 b5c2202b3b provision: add dependency for building pgroonga from source.
clang-9 and llvm-9-dev are added for building pgroonga from source.
2020-03-25 16:09:59 -07:00
arpit551 f23a435fcb circleci: Add base image for Ubuntu Focal.
Added base image for Focal and new job in config.yml
to run backend and frontend tests in circleci for
python3.8 using Focal docker image.
2020-03-25 16:09:59 -07:00
arpit551 b9c8c185bc circleci: Removed python-dev from dockerfile.template.
python-dev will be depreciated in Focal but can be used as python2-dev
so removed it from common dockerfile.template and added it
as an extra package in .circleci/config.yml.
2020-03-25 16:09:59 -07:00
arpit551 0c821424cd lint: Run whitespace linter against .yml files.
Lint now checks for .yml files with whitespace_rules.
2020-03-24 15:31:59 -07:00
Anders Kaseorg 39f9abeb3f python: Convert json.loads(f.read()) to json.load(f).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-24 10:46:32 -07:00
Tim Abbott 3d0b720543 provision: Fix missing apt update for installing new dependencies.
Previously, we only did apt updates when our sources.list files or
keys changed, which could result in provisioning errors for
development systems that don't routinely update their apt cache
(probably including ~all Vagrant environments).
2020-03-23 13:47:17 -07:00
arpit551 dc6c362e4a api_docs: Replaced cgi.escape with html.escape.
cgi.escape is deprecated in python3.2 and removed in python3.8.
This function was unsafe because quote is false by default, hence
removed and replaced with a safer html.escape.
2020-03-23 10:28:21 -07:00
arpit551 258de497a8 setup_venv: Used get_venv_dependencies() to return correct VENV dependencies.
Used get_venv_dependencies function to return the correct dependencies
for RHEL, Centos, Fedora rather than importing them as separate
COMMON_YUM_DEPENDENCIES in provision and create-production-venv.
2020-03-23 10:28:21 -07:00
Dinesh dacca3a593 tests: Add `test-js-with-puppeteer` similar to `test-js-with-casper`.
Note that there are no puppeteer tests yet; this is just preparatory
infrastructure.
2020-03-22 11:42:14 -07:00
Dinesh 9e65f3f354 provision: Add provision support for puppeteer. 2020-03-22 11:42:14 -07:00
Steve Howell eb9a252ec9 populate_db, tests: Restrict emails in zulip realm.
We now restrict emails on the zulip realm, and now
`email` and `delivery_email` will be different for
users.

This change should make it more likely to catch
errors where we leak delivery emails or use the
wrong field for lookups.
2020-03-19 16:21:31 -07:00
Rohitt Vashishtha b25e02aa14 tests: Treat github.com/zulip links as external.
Tests for these links often result in rate-limiting from GitHub,
leading to the builds failing in Circle CI. We temporarily mark
github.com/zulip links as external to keep the builds passing.
2020-03-19 09:27:40 -04:00
Vishnu KS a48845c8fb tests: Create new test mode for generating stripe fixtures. 2020-03-17 16:46:56 -07:00
arpit551 e916d0b733 provision: Added provision support for Ubuntu 20.04(Focal).
Groonga does not have a ppa package for Focal yet so pgroonga is
built from the source.
Postgres 12 is used when os_version is 20.04.
2020-03-17 14:57:14 -07:00
arpit551 b0cb493850 setup_venv: Added get_venv_dependencies function.
Added a get_venv_dependencies() function in setup_venv.py which
returns VENV_DEPENDENCIES according to the vendor and os_version.
The reason for adding this function was because python-dev will be
depreciated in Focal but can be used as python2-dev so when adding
support for Focal VENV_DEPENDENCIES should to be os_version dependent.
2020-03-17 14:57:14 -07:00
arpit551 3963b6740a provision: Corrected how scripts/lib/build-* is included for sha1 sum check.
There were two problems with the previous code-
1) The code glob.glob("scripts/lib/build-") should be
   glob.glob("scripts/lib/build-*) otherwise it would always return [].
2) The part of the code where we included scripts/lib/build-* for sha1 sum
   check would only run when debian is not in os_families(). This wasn't
   correct as we could have a situation where we have to build pgroonga
   from source even in case of debian and so we need to improve the
   condition on it.
Now since we only have build-pgroonga there its better to just directly hash
its content with the condition of BUILD_PGROONGA_FROM_SOURCE.
2020-03-17 14:57:14 -07:00
Mateusz Mandera d00a579318 Revert "test-run-dev: Use a pipe rather than polling a log file."
This should fix spurious failures, where test-run-dev would occasionally
freeze. What exactly about these changes was causing that is still to
be potentially investigated. This is merely meant as a fix to the
failures.
This reverts commit 19429c3ad7.
2020-03-17 14:16:18 -07:00
Vishnu KS 1c6435d4cc validator: Optionally record a type_structure attribute.
We plan to use these records to check and record the schema of Zulip's
events for the purposes of API documentation.

Based on an original messier commit by tabbott.

In theory, a nicer version of this would be able to work directly off
the mypy type system, but this will be good enough for our use case.
2020-03-06 17:07:14 -08:00
Steve Howell e67be55152 node tests: Actually test `translations.js`.
Before this test, we were validating the behavior
of `i18next`, but we weren't validating our light
layer that sits on top of `i18next`, which currently
resides in the slightly misnamed `translations.js`
file.

The translations module is now so small that I'll
just quote it verbatim here:

    import i18next from 'i18next';

    i18next.init({
        lng: 'lang',
        resources: {
            lang: {
                translation: page_params.translation_data,
            },
        },
        nsSeparator: false,
        keySeparator: false,
        interpolation: {
            prefix: "__",
            suffix: "__",
        },
        returnEmptyString: false,  // Empty string is not a valid translation.
    });

    window.i18n = i18next;

We now just do `zrequire('translations')` to initialize
the `i18next` library, which allows us to have simpler
test setup and to actually exercise the above call to
`i18next.init`.

This change now gives us 100% line coverage of `translations.js`,
which of course isn't that hard to acheive (see above).
2020-02-29 12:19:51 -08:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Anders Kaseorg 1cdab5ae61 emoji: Resolve emoji sprite sheets and stylesheets through Webpack.
This gives them cache-compatible URLs, and also avoids some extra
copies of the sprite sheet images.

Comments on the Octopus emoji added by tabbott.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:43:46 -08:00
Anders Kaseorg 197084ab93 generate_emoji_names_table: Update SORTED_CATEGORIES.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:30:31 -08:00
Anders Kaseorg b474fa0e37 webpack: Generalize debug-require-webpack-plugin interface.
Now the caller simply imports the debug ‘require’ function as a
module, deciding for itself how to expose it and with what name (in
our case, we expose it as ‘require’ with expose-loader).  Also, remove
a stray console.log.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:01:06 -08:00
Anders Kaseorg 46e14d1c40 webpack: Expose a version of require() for use in the browser console.
This adds a global require() function that makes JS modules accessible
to the browser console without adding them to the global window
object:

» const typeahead = require("./static/shared/js/typeahead");
» typeahead.popular_emojis
Array(6) [ "1f44d", "1f389", "1f642", "2764", "1f6e0", "1f419" ]

The list of known modules is exposed via the keys of require.ids
object.

This will allow us to migrate more modules to ES6 without losing
access to this debugging functionality.

I’ll probably upload this plugin to NPM at some point, but I figured
I’ll let it bake in-tree first.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-24 23:18:05 -08:00
Steve Howell 5e8279c2fb refactor: Extract settings_config.
This moves some code from settings_display.js
into the new module settings_config.js.

Extracting this module breaks some dependencies
on settings_display.js (which has some annoying
transitive dependencies, including jQuery).

In particular this isolates stream_data from
from settings_display.js.

Two of the three structures that we moved here
weren't even directly used by settings_display.js,
since we do a lot of rendering in the modules
admin.js and setting.js.

We make get_all_display_settings() a function
to avoid a require-time dependency on page_params.

Breaking the dependencies simplifies a few
node tests.

Most of the node test complexity came from the
following commit in March 2019:

5a130097bf

The commit itself seems harmless enough, but
dependencies can have a somewhat "viral" nature,
where making stream_data depend on settings_display
caused us to modify four different node tests.
2020-02-21 12:06:31 -08:00
Tim Abbott 8aae02de68 emoji: Add a block comment explaining spritesheet CSS. 2020-02-21 10:26:57 -08:00
Steve Howell af7923c557 refactor: Extract get_square_size() for emoji sprites.
This cleans up a few things:

    - just yield values so we don't have to do
      tedious max logic
    - use values() instead of items() for
      skin_variations loop

In the ideal world the emoji.json would reduce this
code to `get_square_size = lambda data: data['square_size']`,
but I don't think we can get the square size explicitly.
2020-02-21 10:21:12 -08:00
Steve Howell da1ce9a577 emoji sprites: Avoid hard coding CSS percentages.
This commit changes the calculation of the
background-size parameter that we use to
render emojis from sprite sheets.

In particular, it now makes the parameter
match the sizes of our latest sprite
sheets from Twitter/Google.

This should fix the geometry aspect of #13959,
but we also need to fix some issues with the
cache being sticky.

There is also some minor cleanup:

    - Remove obsolete -moz/-webkit CSS.
    - Remove needless precision in percentages.
    - Fix the transposed nrows/ncols names.
    - Add extensive commenting.

Finally, we add a minor bump to the provision
number.  This commit should be merged in the
same series as the other fix for this issue,
which will probably have a major bump, and we'll
need to rebase this appropriately.
2020-02-21 10:21:12 -08:00
Anders Kaseorg 9d598d95a6 puppet: Fix puppet-lint warning.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-20 19:51:48 -08:00
Tim Abbott aaee506fb2 provision: Always build pygments_data.json if missing.
While it's a bit of extra complexity to do this check, which I'm not
excited about, we've had multiple folks spend significant time being
confused rebasing past d7d8632525 into
deleting `pygments_data.json`, with provision not rebuilding it, so
this seems worth merging as a transitional fix even if we decide to
remove it in 2 months.
2020-02-20 10:25:11 -08:00
Mateusz Mandera 7293862eae tests: Keep temporary .coverage files in var/ directory. 2020-02-19 14:36:17 -08:00
Mateusz Mandera 782353f0e1 tests: Ensure coverage data files get cleaned up at the end.
Without calling cov.erase() the data file seems to persist and even
pollute future test runs if not removed. Registering an atexit handler
seems like a good, and reasonably clean way to ensure the cleanup
happens.
Fixes #13933.
2020-02-19 14:36:17 -08:00
Anders Kaseorg 7990ef2d52 register: Spell Greek Ακμή with Greek alpha, not Latin A
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-18 17:41:43 -08:00
Anders Kaseorg 4889a0486d tests: Compile Handlebars templates with source maps.
This allows us to collect coverage for Handlebars templates, and also
improves the readability of Handlebars-related stack traces.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-18 07:38:46 -05:00
Mateusz Mandera 36a8e61e67 run-dev: Run process_queue with DJANGO_AUTORELOAD_ENV.
In Django 2.2 the autoreload system has changed.
DJANGO_AUTORELOAD_ENV env variable should be set when calling code
that'll use the autoreloader. Otherwise there's some kind of race
condition in the autoreload code when SIGINT is sent, where
restart_with_reloader() (called only if the env variable isn't set)
has the subprocess module calling p.kill() on a process that's already
exited, raising ProcessLookupError and printing an ugly traceback. This
causes non-deterministic test-run-dev failures.
2020-02-17 13:06:50 -05:00
Vishnu KS 5bab2a3762 upload: Replace jQuery filedrop with Uppy. 2020-02-13 16:43:19 -08:00
Tim Abbott e2892a88c1 channel: Discard all HTTP responses while reloading.
We used to have a block of code doing this just in the presence
endpoint because that's where we'd had error-handling problems with it
not being present, but it seems more correct for it to run
unconditionally on all HTTP requests.

This requires adding a dependency of channel on reload_state, which we
record in the webpack configuration for now.
2020-02-13 15:45:39 -08:00
Tim Abbott 906160f1a3 presence: Re-introduce data filtering when offline.
This should return us to a situation where we won't get blueslip
browser error reporting for users created while a device was offline
just before it reloads.
2020-02-13 15:45:39 -08:00
Hashir Sarwar eb23c6fa6c test_fixtures: Clean up interface for `template_database_status()`.
1) Created a new class `DatabaseType` and access its objects inside
`template_database_status()` instead of sending five arguments with
default values.

2) Made `check_files` and `setting_name` local variables instead of
function parameters since they had same value(None) for every call.

Fixes #13845.
2020-02-12 11:07:10 -08:00
Anders Kaseorg d7d8632525 pygments_data: Replace JS module with JSON module.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:09:12 -08:00
Anders Kaseorg e257253e64 emoji_codes: Replace JS module with JSON module.
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.

Update the backend to use emoji_codes.json too instead of the three
separate JSON files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:09:12 -08:00
Steve Howell 7ec5fbab2e node tests: Remove templates test.
I believe we can remove these and rely on
other parts of our testing/code-review
to ensure template quality.

These tests never really exercised our
app code, as evidenced by us not regressing
any of the 100%-line-coverage files.

We have a couple other ways that we verify
the correct format of the templates:

    - webpack (can they compile?)
    - check-templates (are they nicely indented?)

For deep testing, we have Casper, which
exercises most of our most important templates
in some meaningful way.

I think it's pretty rare that we get bugs
now that are directly caused by bad templates,
and an even smaller subset of them would
have been caught by the node tests.

If that trend changes in the future, I would prefer to
just do something "greenfield" to address
any common problems rather than resurrect
this code, but we could always resurrect it
from git.

The template node tests did check a little bit of
detail about which fields are there, but not
in an integrated way, so that aspect of the tests
wasn't very useful either.
2020-02-12 09:58:23 -08:00
Chris Heald 2ca447c1a5
provisioning: Support Ubuntu 19.10 eoan.
This adds Ubuntu 19.10 as a valid provisioning target.

The release test in setup-apt-repo was changed from a list of values to
a regex check for brevity.
2020-02-12 09:56:42 -08:00
Greg Price 1e2b558b01 reset-to-pull-request: Add a usage message.
Every CLI program should have a usage message.

Also add a mention in the `push-to-pull-request` usage message of
its participation in the `refs/remotes/pr/` pseudo-remote feature.
2020-02-11 14:45:17 -08:00
Greg Price e3d843baf9 push-to-pull-request: Give a more detailed error on URL parse failure.
This simplifies debugging when this error fires, e.g. for finding
an issue like the one we just used `git remote get-url` to fix.
2020-02-11 14:45:17 -08:00
Greg Price e7a82e732e push-to-pull-request: Use `git remote get-url`.
This gives us the right behavior when using the `url.*.insteadOf`
mechanism for aliases in Git remote URLs.  For example, if
one's ~/.gitconfig has:

  [url "git@github.com:"]
      insteadOf = gh:

then `git remote add upstream gh:zulip/zulip` will work great, as
the nice, short, mnemonic `gh:` prefix gets expanded to the more
finicky `git@github.com:`.  I use just such a prefix routinely.

But the feature does require that scripts go through the right
abstractions.  In particular `git remote get-url`, since Git 2.7
(from 2016), exists for exactly this reason.  A plain `git config`
command bypasses the expansion, getting the verbatim `gh:...`
version, which doesn't work.

So, switch to that.

As a bonus, we get to behave correctly if for some reason the user
has configured a push URL distinct from the fetch URL for this
remote, just by adding `--push`.  With `git config`, we'd have had
to manually implement the fallback from `remote.upstream.pushUrl` to
`remote.upstream.url` in order to properly handle that case.
2020-02-11 14:45:17 -08:00
Steve Howell 6922eef380 signups: Use internal_send_stream_message().
We prefer this to internal_send_message().

We are trying to deprecate `internal_send_message`,
which has extra moving parts related to
`extract_recipients` and `Addressee.legacy_build`.

There are two chunks of code that I touch here
that look pretty similar, but I'm not quite
sure they're worth de-duplicating, since they
use different topics and different message
content.
2020-02-10 15:45:13 -08:00
Steve Howell b33552997e cross realm bots: Simplify notify_new_user.
Instead of having `notify_new_user` delegate
all the heavy lifting to `send_signup_message`,
we just rename `send_signup_message` to be
`notify_new_user` and remove the one-line
wrapper.

We remove a lot of obsolete complexity:

    - `internal` was no longer ever set to True
      by real code, so we kill it off as well
      as well as killing off the internal_blurb code
      and the now-obsolete test

    - the `sender` parameter was actually an
      email, not a UserProfile, but I think
      that got past mypy due to the caller
      passing in something from settings.py

    - we were only passing in NOTIFICATION_BOT
      for the sender, so we just hard code
      that now

    - we eliminate the verbose
      `admin_realm_signup_notifications_stream`
      parameter and just hard code it to
      "signups"

    - we weren't using the optional realm
      parameter

There's also a long ugly comment in
`get_recipient_info` related to this code
that I amended for now.
We should try to take action in a subsequent
commit.
2020-02-10 15:45:13 -08:00
Anders Kaseorg 719546641f js: Convert a.indexOf(…) !== -1 to a.includes(…).
Babel polyfills this for us for Internet Explorer.

import * as babelParser from "recast/parsers/babel";
import * as recast from "recast";
import * as tsParser from "recast/parsers/typescript";
import { builders as b, namedTypes as n } from "ast-types";
import K from "ast-types/gen/kinds";
import fs from "fs";
import path from "path";
import process from "process";

const checkExpression = (node: n.Node): node is K.ExpressionKind =>
  n.Expression.check(node);

for (const file of process.argv.slice(2)) {
  console.log("Parsing", file);
  const ast = recast.parse(fs.readFileSync(file, { encoding: "utf8" }), {
    parser: path.extname(file) === ".ts" ? tsParser : babelParser,
  });
  let changed = false;

  recast.visit(ast, {
    visitBinaryExpression(path) {
      const { operator, left, right } = path.node;
      if (
        n.CallExpression.check(left) &&
        n.MemberExpression.check(left.callee) &&
        !left.callee.computed &&
        n.Identifier.check(left.callee.property) &&
        left.callee.property.name === "indexOf" &&
        left.arguments.length === 1 &&
        checkExpression(left.arguments[0]) &&
        ((["===", "!==", "==", "!=", ">", "<="].includes(operator) &&
          n.UnaryExpression.check(right) &&
          right.operator == "-" &&
          n.Literal.check(right.argument) &&
          right.argument.value === 1) ||
          ([">=", "<"].includes(operator) &&
            n.Literal.check(right) &&
            right.value === 0))
      ) {
        const test = b.callExpression(
          b.memberExpression(left.callee.object, b.identifier("includes")),
          [left.arguments[0]]
        );
        path.replace(
          ["!==", "!=", ">", ">="].includes(operator)
            ? test
            : b.unaryExpression("!", test)
        );
        changed = true;
      }
      this.traverse(path);
    },
  });

  if (changed) {
    console.log("Writing", file);
    fs.writeFileSync(file, recast.print(ast).code, { encoding: "utf8" });
  }
}

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-10 14:08:12 -08:00
Steve Howell 3af8dc6c7d node tests: Add some color to output. 2020-02-09 22:07:51 -08:00
Steve Howell e1977f4680 node tests: Use EXEMPT_FILES for line coverage.
We now have 100% line coverage on 71 JS files.
This is thanks to about 150 people who have
contributed code to frontend/node_tests.

And then 126 files are still short of 100% line
coverage.

We now enforce line coverage with a set called
EXEMPT_FILES, which are the files for which
we do NOT expect to have 100% coverage.

Using an exemption list makes it so that adding
a new JS file to the project without 100% line
coverage will cause the build to fail.  This will
encourage folks to be intentional about their
lack of test coverage.
2020-02-09 22:07:51 -08:00
Steve Howell fe48ede9cf node tests: Fix bug with zero-coverage case.
If a file that had 100% coverage somehow regressed
to 0% coverage, we would report an error to the
console, but we weren't treating it as an actual
failure.

We've probably always had this bug, but it probably
rarely was an issue, since devs might have seen
the error locally, or hopefully whatever crazy
thing you did to totally remove coverage would
have had other symptoms.

If this was intentional, I suspect it might have
had something to do with wanting to get coverage
reports when you just run individual tests.  But
a while back we changed it so that when you run
individual tests, we don't do the line coverage
enforcement.
2020-02-09 22:07:51 -08:00
Vishnu KS 4572be8c27 api: Rename subject_links to topic_links.
Fixes #13588
2020-02-07 14:35:22 -08:00
Anders Kaseorg 02511bff1c js: Automatically convert _.each to for…of.
This commit was automatically generated by the following script,
followed by lint --fix and a few small manual lint-related cleanups.

import * as babelParser from "recast/parsers/babel";
import * as recast from "recast";
import * as tsParser from "recast/parsers/typescript";
import { builders as b, namedTypes as n } from "ast-types";
import { Context } from "ast-types/lib/path-visitor";
import K from "ast-types/gen/kinds";
import { NodePath } from "ast-types/lib/node-path";
import assert from "assert";
import fs from "fs";
import path from "path";
import process from "process";

const checkExpression = (node: n.Node): node is K.ExpressionKind =>
  n.Expression.check(node);
const checkStatement = (node: n.Node): node is K.StatementKind =>
  n.Statement.check(node);

for (const file of process.argv.slice(2)) {
  console.log("Parsing", file);
  const ast = recast.parse(fs.readFileSync(file, { encoding: "utf8" }), {
    parser: path.extname(file) === ".ts" ? tsParser : babelParser,
  });
  let changed = false;
  let inLoop = false;
  let replaceReturn = false;

  const visitLoop = (...args: string[]) =>
    function(this: Context, path: NodePath) {
      for (const arg of args) {
        this.visit(path.get(arg));
      }
      const old = { inLoop };
      inLoop = true;
      this.visit(path.get("body"));
      inLoop = old.inLoop;
      return false;
    };

  recast.visit(ast, {
    visitDoWhileStatement: visitLoop("test"),

    visitExpressionStatement(path) {
      const { expression, comments } = path.node;
      let valueOnly;
      if (
        n.CallExpression.check(expression) &&
        n.MemberExpression.check(expression.callee) &&
        !expression.callee.computed &&
        n.Identifier.check(expression.callee.object) &&
        expression.callee.object.name === "_" &&
        n.Identifier.check(expression.callee.property) &&
        ["each", "forEach"].includes(expression.callee.property.name) &&
        [2, 3].includes(expression.arguments.length) &&
        checkExpression(expression.arguments[0]) &&
        (n.FunctionExpression.check(expression.arguments[1]) ||
          n.ArrowFunctionExpression.check(expression.arguments[1])) &&
        [1, 2].includes(expression.arguments[1].params.length) &&
        n.Identifier.check(expression.arguments[1].params[0]) &&
        ((valueOnly = expression.arguments[1].params[1] === undefined) ||
          n.Identifier.check(expression.arguments[1].params[1])) &&
        (expression.arguments[2] === undefined ||
          n.ThisExpression.check(expression.arguments[2]))
      ) {
        const old = { inLoop, replaceReturn };
        inLoop = false;
        replaceReturn = true;
        this.visit(
          path
            .get("expression")
            .get("arguments")
            .get(1)
            .get("body")
        );
        inLoop = old.inLoop;
        replaceReturn = old.replaceReturn;

        const [right, { body, params }] = expression.arguments;
        const loop = b.forOfStatement(
          b.variableDeclaration("let", [
            b.variableDeclarator(
              valueOnly ? params[0] : b.arrayPattern([params[1], params[0]])
            ),
          ]),
          valueOnly
            ? right
            : b.callExpression(
                b.memberExpression(right, b.identifier("entries")),
                []
              ),
          checkStatement(body) ? body : b.expressionStatement(body)
        );
        loop.comments = comments;
        path.replace(loop);
        changed = true;
      }
      this.traverse(path);
    },

    visitForStatement: visitLoop("init", "test", "update"),

    visitForInStatement: visitLoop("left", "right"),

    visitForOfStatement: visitLoop("left", "right"),

    visitFunction(path) {
      this.visit(path.get("params"));
      const old = { replaceReturn };
      replaceReturn = false;
      this.visit(path.get("body"));
      replaceReturn = old.replaceReturn;
      return false;
    },

    visitReturnStatement(path) {
      if (replaceReturn) {
        assert(!inLoop); // could use labeled continue if this ever fires
        const { argument, comments } = path.node;
        if (argument === null) {
          const s = b.continueStatement();
          s.comments = comments;
          path.replace(s);
        } else {
          const s = b.expressionStatement(argument);
          s.comments = comments;
          path.replace(s, b.continueStatement());
        }
        return false;
      }
      this.traverse(path);
    },

    visitWhileStatement: visitLoop("test"),
  });

  if (changed) {
    console.log("Writing", file);
    fs.writeFileSync(file, recast.print(ast).code, { encoding: "utf8" });
  }
}

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-07 14:09:47 -08:00
Anders Kaseorg 811e128787 check-openapi: Fix lint errors and remove lint exclusion.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-07 14:09:47 -08:00
Anders Kaseorg b3f63262af lint: Delegate console.log check to ESLint.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-07 14:09:47 -08:00
Anders Kaseorg 61bf698185 requirements: Fork pip.txt to pip2.txt for Python 2.
The current version of setuptools no longer supports Python 2.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg d2e07ea51b setup_venv: Replace virtualenv_args list with python2 bool.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg fa558de6d8 commit-message-lint: Detect the upstream remote name.
My upstream remote is named origin, so commit-message-lint was always
complaining at me.  Detect the right remote name from the output of
`git remote -v`.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-04 13:57:51 -08:00
Mateusz Mandera 4c5a8e6f0c queue: Remove missedmessage_email_senders. 2020-01-31 12:13:51 -08:00
Mateusz Mandera a583da67b3 test-queue-worker-reload: Fix expected number of workers. 2020-01-31 12:13:51 -08:00
Steve Howell eeee6edf41 pm_list: Simplify redraws for Private Messages.
We now use vdom-ish techniques to track the
list items for the pm list.  When we go to update
the list, we only re-render nodes whose data
has changed, with two exceptions:

    - Obviously, the first time we do a full render.
    - If the keys for the items have changed (i.e.
      a new node has come in or the order has changed),
      we just re-render the whole list.

If the keys are the same since the last re-render, we
only re-render individual items if their data has
changed.

Most of the new code is in these two modules:

    - pm_list_dom.js
    - vdom.js

We remove all of the code in pm_list.js that is
related to updating DOM with unread counts.

For presence updates, we are now *never*
re-rendering the whole list, since presence
updates only change individual line items and
don't affect the keys.  Instead, we just update
any changed elements in place.

The main thing that makes this all work is the
`update` method in `vdom`, which is totally generic
and essentially does a few simple jobs:

    - detect if keys are different
    - just render the whole ul as needed
    - for items that change, do the appropriate
      jQuery to update the item in place

Note that this code seems to play nice with simplebar.

Also, this code continues to use templates to render
the individual list items.

FWIW this code isn't radically different than list_render,
but it's got some key differences:

    - There are fewer bells and whistles in this code.
      Some of the stuff that list_render does is overkill
      for the PM list.

    - This code detects data changes.

Note that the vdom scheme is agnostic about templates;
it simply requires the child nodes to provide a render
method.  (This is similar to list_render, which is also
technically agnostic about rendering, but which also
does use templates in most cases.)

These fixes are somewhat related to #13605, but we
haven't gotten a solid repro on that issue, and
the scrolling issues there may be orthogonal to the
redraws.  But having fewer moving parts here should
help, and we won't get the rug pulled out from under
us on every presence update.

There are two possible extensions to this that are
somewhat overlapping in nature, but can be done
one a time.

    * We can do a deeper vdom approach here that
      gets us away from templates, and just have
      nodes write to an AST.  I have this on another
      branch, but it might be overkill.

    * We can avoid some redraws by detecting where
      keys are moving up and down.  I'm not completely
      sure we need it for the PM list.

If this gets merged, we may want to try similar
things for the stream list, which also does a fairly
complicated mixture of big-hammer re-renders and
surgical updates-in-place (with custom code).

BTW we have 100% line coverage for vdom.js.
2020-01-30 13:11:32 -08:00
Mateusz Mandera a25f00a69c populate_db: Extract some functions to server_initialization.py. 2020-01-29 17:26:45 -08:00
Tim Abbott 7bf3312114 api: Document new get_messages oldest/newest API feature.
While we're at it, we make the examples more sensible.
2020-01-29 12:14:06 -08:00
Steve Howell b945cfd291 test-js-with-node: Extract run_tests_via_node_js(). 2020-01-27 16:32:11 -08:00
Steve Howell cc07bf04b7 test-js-with-node: Move constants toward top. 2020-01-27 16:32:11 -08:00
Steve Howell ed845d4da4 minor: Add static/shared/js to not-enforced logic.
This is an expedient fix--we should possibly just
require all modules in static/shared to be 100%
covered.
2020-01-27 16:32:11 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
Steve Howell f24de074d4 node tests: Get 100% coverage on topic_list_data. 2020-01-22 14:31:33 -08:00
Mateusz Mandera 8dd95bd057 tests: Replace httpretty with responses.
responses is an module analogous to httpretty for mocking external
URLs, with a very similar interface (potentially cleaner in that it
makes use of context managers).

The most important (in the moment) problem with httpretty is that it
breaks the ability to use redis in parts of code where httpretty is
enabled.  From more research, the module in general has tendency to
have various troublesome bugs with breaking URLs that it shouldn't be
affecting, caused by it working at the socket interface layer.  While
those issues could be fixed, responses seems to be less buggy (based
on both third-party reports like ckan/ckan#4755 and our own experience
in removing workarounds for bugs in httpretty) and is more actively
maintained.
2020-01-22 11:56:15 -08:00
Aditya Bansal d0492b103b template_parser: Fix Jinja2 tag matching with strip whitespace syntax.
In this commit, we basically match any kinda of jinja2 start tag,
    no matter its special kind (eg. jinja2_whitespace_stripped_start)
    to any kinda jinja2 end tag (eg. jinja2_whitespace_stripped_end)

    Idea is special operators like `-` do not change the meaning of
    inline tag and thus matching shouldn't depend upon this.
2020-01-16 12:53:34 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Steve Howell 752d6dc6df tools: Remove find-add-class tool.
I added this tool a few years ago, and I did have
a vision for how it would improve our codebase, but
I can't remember exactly where I was going with it.

At this point the tool is just a little too noisy
to be helpful.  An example of it creating confusion
was a recent PR where somebody was patching
user_circle_class in the PM list, and we already
had similar code in the buddy list, because they
use the same CSS.  I mean, there was possibly a way
that the code could have been structured to remove
some of the duplication, but it probably would have
just moved the complexity around.

I just don't think it's worth maintaining the tool
at this point.
2020-01-14 15:45:49 -08:00
Steve Howell 29e63c0417 Fix type errors in LazySet.
I think the only place that was broken is where
we copy users from streams.
2020-01-14 15:40:40 -08:00
Tim Abbott 79f18138f5 realm: Add private_message_policy setting.
This experimental setting disables sending private messages in Zulip
in a crude way (i.e. users get an error when they try to send one).
It makes no effort to adjust the UI to avoid advertising the idea of
sending private messages.

Fixes #6617.
2020-01-13 12:20:42 -08:00
rht 60563e9180 provision: Install postgresql-devel on CentOS as system dependency. 2020-01-07 13:25:25 -08:00
Mateusz Mandera c011d2c6d3 email_mirror: Migrate missed message addresses from redis to database.
Addresses point 1 of #13533.

MissedMessageEmailAddress objects get tied to the specific that was
missed by the user. A useful benefit of that is that email message sent
to that address will handle topic changes - if the message that was
missed gets its topic changed, the email response will get posted under
the new topic, while in the old model it would get posted under the
old topic, which could potentially be confusing.

Migrating redis data to this new model is a bit tricky, so the migration
code has comments explaining some of the compromises made there, and
test_migrations.py tests handling of the various possible cases that
could arise.
2020-01-07 13:03:22 -08:00
Steve Howell 94761b806c node tests: Restore 100% coverage to pm_list. 2020-01-06 10:21:23 -08:00
Steve Howell da1392efd2 node test: Remove complicated pm_list test.
This test mostly tests logic that I'm about
to remove in subsequent commits, and it's a bit
messy.

This commit removes 100% line coverage, but I
will restore that a few commits later.
2020-01-06 10:21:23 -08:00
Steve Howell 4e59937632 js: Add IntDict class.
We don't use this yet, but we will soon.

We report errors if users pass in strings instead of
ints, but we try to still use the key.
2020-01-05 12:27:26 -08:00
Steve Howell ee3e488e02 js: Extract FoldDict class.
We have ~5 years of proof that we'll probably never
extend Dict with more options.

Breaking the classes into makes both a little faster
(no options to check), and we remove some options
in FoldDict that are never used (from/from_array).

A possible next step is to fine-tune the Dict to use
Map internally.

Note that the TypeScript types for FoldDict are now
more specific (requiring string keys).  Of course,
this isn't really enforced until we convert other
modules to TS.
2020-01-03 17:19:50 -08:00
Mateusz Mandera d691c249db api: Return a JsonableError if API key of invalid format is given. 2020-01-03 16:56:42 -08:00
Anders Kaseorg 1f8d21af74 test-install: Use lxc-destroy -f instead of lxc-stop.
Fixes this error after rebooting the host:

$ sudo ./destroy-all  -f
zulip-install-bionic-41MM2
lxc-stop: zulip-install-bionic-41MM2: tools/lxc_stop.c: main: 191 zulip-install-bionic-41MM2 is not running

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-18 03:48:39 -08:00
Anders Kaseorg 9b5f9858fb test-install: Run lxc-attach with --clear-env.
The host environment variables (especially PATH) should not be allowed
to pollute the test and could interfere with it.

This allows test-install to run on a NixOS host.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-18 03:48:39 -08:00
Anders Kaseorg ab211c7acf lint: Tell ShellCheck to look for sourced files at relative paths.
This uses the new -P option of ShellCheck 0.7.0.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-18 03:48:02 -08:00
Tim Abbott 3dc19344c0 renumber-migrations: Exclude migration 0209.
Otherwise, this tries to renumber the already merged duplicate 0209s,
which isn't helpful.
2019-12-11 16:08:32 -08:00
Mateusz Mandera 586a5facc9 models: Add is_realm_admin and is_guest setters.
Fixes #13452.

The migration from UserProfile.is_realm_admin/UserProfile.is_guest in
e10361a832 broke our LDAP-based support
for setting a user's role via LDAP properties, which relied on setting
those fields.  Because the django-auth-ldap feature powering that only
supports booleans (and in any case, we don't want to expose constants
like `ROLE_REALM_ADMINISTRATOR` to the LDAP configuration interface),
it makes sense to provide setters for these legacy fields for
backwards-compatibility.

We lint against using these setters directly in Zulip's codebase
directly.  The issue with using these is that when changing user's
.role we want to create appropriate RealmAuditLog entries and send
events. This isn't possible when using these setters - the log entries
and events should be created if the role change in the UserProfile is
actually save()-ed to the database - and on the level of the setter
function, it's not known whether the change will indeed be saved.

It would have to be somehow figured out on the level of post_save
signal handlers, but it doesn't seem like a good design to have such
complexity there, for the sake of setters that generally shouldn't be
used anyway - because we prefer the do_change_is_* functions.

The purpose of this change is narrowly to handle use cases like the
setattr on these boolean properties.
2019-12-09 11:54:01 -08:00
Anders Kaseorg f15cfc00ef test-documentation: Invoke sphinx-build via make instead of directly.
This lets us pick up the -W option from docs/Makefile.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-02 18:39:33 -08:00
Anders Kaseorg fdaca911e4 docs: Replace Makefile based on output of newer sphinx-quickstart.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-12-02 18:39:33 -08:00
Mateusz Mandera 0c2cc41d2e CVE-2019-18933: Fix insecure account creation via social authentication.
A bug in Zulip's new user signup process meant that users who
registered their account using social authentication (e.g. GitHub or
Google SSO) in an organization that also allows password
authentication could have their personal API key stolen by an
unprivileged attacker, allowing nearly full access to the user's
account.

Zulip versions between 1.7.0 and 2.0.6 were affected.

This commit fixes the original bug and also contains a database
migration to fix any users with corrupt `password` fields in the
database as a result of the bug.

Out of an abundance of caution (and to protect the users of any
installations that delay applying this commit), the migration also
resets the API keys of any users where Zulip's logs cannot prove the
user's API key was not previously stolen via this bug.  Resetting
those API keys will be inconvenient for users:

* Users of the Zulip mobile and terminal apps whose API keys are reset
  will be logged out and need to login again.
* Users using their personal API keys for any other reason will need
  to re-fetch their personal API key.

We discovered this bug internally and don't believe it was disclosed
prior to our publishing it through this commit.  Because the algorithm
for determining which users might have been affected is very
conservative, many users who were never at risk will have their API
keys reset by this migration.

To avoid this on self-hosted installations that have always used
e.g. LDAP authentication, we skip resetting API keys on installations
that don't have password authentication enabled.  System
administrators on installations that used to have email authentication
enabled, but no longer do, should temporarily enable EmailAuthBackend
before applying this migration.

The migration also records which users had their passwords or API keys
reset in the usual RealmAuditLog table.
2019-11-21 10:23:37 -08:00
Anders Kaseorg f9f104a4f8 js: Automatically convert var to let and const in more files.
This commit was automatically generated by `tools/lint --only=eslint
--fix`, after an `.eslintrc.json` change.

A half dozen files were removed from the changes by tabbott pending
further work to ensure we avoid breaking valuable PRs with merge
conflicts.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-20 14:10:47 -08:00
Tim Abbott 4e421ebe12 scripts: Move inline-email-css from tools to scripts.
We'll be soon documenting a production workflow that involves using
it, and that means it needs to live under scripts/ (since tools/ isn't
present in release tarballs).
2019-11-15 17:39:42 -08:00
Vishnu Ks 00455df7f9 tests: Use admin client for curl examples test. 2019-11-15 15:53:31 -08:00
Vishnu Ks bb5463b29a tests: Run test_generated_curl_examples_for_success after test_the_api.
Upcoming changes in test_generated_curl_examples_for_success modifies
various data of iago user heavily. So it's much easier to run
test_the_api initially than making various changes in tests of
test_the_api function.
2019-11-15 15:53:31 -08:00
Anders Kaseorg 0ce4de0421 lint: Run mypy with --no-error-summary.
This suppresses the mypy message “Success: no issues found in 1085
source files” or “Found 1 error in 1 file (checked 1085 source files)”
in the output of lint.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 13:26:02 -08:00
Anders Kaseorg 2c4101dbc5 dependencies: Upgrade simplebar from 4.2.3 to 5.0.7.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:46:29 -08:00
Anders Kaseorg 40f4ead738 mypy: Upgrade from 0.720 to 0.730.
Fixes #13269.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 43bba63549 upgrade-python-dependencies: Remove.
This tool doesn’t match our current workflow for Python requirements
upgrades as of commit ec9bf6576a (#13213).
It also has a type error with mypy 0.730, which would be easily fixable,
but removing it is easier.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg 1a622f91f9 list-outdated-packages: Remove.
This tool provides no value over `pip list --outdated`.  It also has a
type error with mypy 0.730, which would be easily fixable, but
removing it is easier.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:38:45 -08:00
Anders Kaseorg cafac83676 request: Tighten type checking on REQ.
Then, find and fix a predictable number of previous misuses.

With a small change by tabbott to preserve backwards compatibility for
sending `yes` for the `forged` field.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-13 12:35:55 -08:00
Anders Kaseorg 5e89a7d646 install-shellcheck: Upgrade ShellCheck from 0.6.0 to 0.7.0.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-11 16:26:31 -08:00
Anders Kaseorg fffef412bc dependencies: Upgrade to-markdown 3.1.1 to turndown 5.0.3.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-11 16:26:31 -08:00
David Rosa b041948132 docs: Reorganize auth and migrations subsystems.
- Moves "Authentication in the development environment" from subsystems
to "development/authentication.md".
- Moves "Renumbering migrations" to a section within "Schema migrations".
2019-11-07 09:42:36 -08:00
Anders Kaseorg 0584a7938f tsconfig: Move to top level.
This way, webpack.config.ts is type checked.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-04 18:12:11 -08:00
Anders Kaseorg de4685441c typescript: Type webpack.config.ts correctly.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-11-04 18:12:11 -08:00
Vishnu KS 8e9ccdf376 tests: Remove get-raw-message from curl test exclude_list. 2019-10-30 16:49:26 -07:00
Anders Kaseorg 042c558bb3 eslint: Enable sort-imports rule.
I figure we should enable this before we have lots of imports.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-30 13:10:25 -07:00
Anders Kaseorg 2bbcd6ab34 bundles: Factor out portico bundle.
This adds translations.js to the digest entrypoint.  Presumably that’s
fine.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:53:15 -07:00
Anders Kaseorg fda70a5458 templates: Use just one webpack entry point per page.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:53:15 -07:00
Anders Kaseorg f245fcf408 webpack: Enable code splitting and deduplication.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:53:15 -07:00
Anders Kaseorg d1a3bf424a requirements: Use webpack4 fork of django-webpack-loader.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:53:15 -07:00
Anders Kaseorg ee9a6071fd 5xx.html: Build with webpack.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:53:15 -07:00
Anders Kaseorg f077508c6d test-backend, run-casper: Remove proxy vars instead of setting to "".
Setting http_proxy to the empty string caused a spurious error from
xmlsec:

>>> import os
>>> os.environ["http_proxy"] = ""
>>> import xmlsec
error : Unknown IO error

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:47:45 -07:00
Anders Kaseorg c3bf7c6889 webpack: Consolidate production and development plugin configuration.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:39:34 -07:00
Anders Kaseorg 27fac76da8 styles: Move media queries into the files they override.
Webpack code splitting will make the inclusion order of CSS files less
obvious, and we need to guarantee that these rules follow the rules
they override.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:39:17 -07:00
Mateusz Mandera 28dd1b34f2 auth: Refactor social login rendering.
login_context now gets the social_backends list through
get_social_backend_dicts and we  move display_logo customization
to backend class definition.

This prepares for easily adding multiple IdP support in SAML
authentication - there will be a social_backend dict for each configured
IdP, also allowing display_name and icon customization per IdP.
2019-10-28 15:06:26 -07:00
Anders Kaseorg 453919bbc7 version: Only let `git describe` match tags beginning with a digit.
This will let us use other tags for things like `@zulip/shared`
versions.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-24 14:54:45 -07:00
David Rosa bd3a8f1e96 docs: Merge front-end-build-process with html-css.
This merges the "Static asset pipeline" article as a new section
at the bottom of the "HTML and CSS" article.
2019-10-23 13:08:36 -07:00
Vishnu KS ec955f8f78 support: Show confirmation links in search.
Fixes #13060 #12784
2019-10-21 16:56:50 -07:00
Vishnu KS 62a8e378a6 support: Refactor realm details into a seperate template. 2019-10-21 16:52:46 -07:00
Vishnu KS f2242c1487 docs: Make update-notification-settings use curl example system. 2019-10-21 17:10:35 +05:30
Greg Price a63786ac0d shared: Set up a way to share some frontend code with the mobile app.
This adds the general machinery required, and sets it up for the file
`typing_status.js` as a first use case.

Co-authored-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-17 16:48:23 -07:00
Anders Kaseorg 173c9cee42 frontend_tests: Switch from ts-node to Babel; add rewire-ts plugin.
This will let tests rewrite TypeScript/ES6 module bindings that would
otherwise be read-only.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-17 16:48:23 -07:00
Tim Abbott fa26475fcf docs: Deal with a few broken links. 2019-10-15 14:03:53 -07:00
Anders Kaseorg dfd9ace7fa requirements: Add back future.
It happens that commonmark, python-jose, and python-twitter don’t
actually use future on Python 3, and moto uses aws-xray-sdk in such a
way that it doesn’t use future, but this was a weird game to be
playing just to remove one dependency, and it caused CI failures after
new releases of future, so let’s just include it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-11 14:04:35 -07:00
Vishnu KS 1585ad7bf4 mattermost: Add support for exporting DMs and huddles. 2019-10-10 16:37:03 -07:00
Tim Abbott f8928182cf run-dev: Set HTTP header to show we're proxing from port 9991.
Previously, while Django code that relied on EXTERNAL_HOST and other
settings would know the Zulip server is actually on port 9991, the
upcoming Django SAML code in python-social-auth would end up detecting
a port of 9992 (the one the Django server is actually listening on).
We fix this using X-Forwarded-Port.
2019-10-08 17:53:09 -07:00
Rishi Gupta 360cd7f147 remote data: Send RealmAuditLog data. 2019-10-08 17:27:29 -07:00
Ray Kraesig 1180299fe4 provisioning: fix up bash profile creation
The base Ubuntu image includes no `.bash_profile`. Creating one will
prevent the existing support scripts in `~/.profile` from loading.

Instead, append our setup code to [whichever script Bash will load
first][bash-startup].

[bash-startup]: https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html#Bash-Startup-Files
2019-10-08 17:17:55 -07:00
Anders Kaseorg 3d876aacc6 requirements: Use pip-compile --generated-hashes for better security.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:21:18 -07:00
Anders Kaseorg 9f09d4fcfd update-locked-requirements: Print some progress information.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-06 15:21:18 -07:00
Anders Kaseorg 0af22dad18 flush-memcached: Respect MEMCACHED_LOCATION; handle errors.
Fixes #13238.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-01 16:05:55 -07:00
Tim Abbott 23c48764b5 README: Improve links to coverage/CI to point to master.
Apparently, the CircleCI and Codecov links (and the Codecov badge)
weren't pointing specifically at master, so they'd sometimes show
state from the lastest push to a pull request, which isn't a
reasonable way to advertise whether the project's build is passing.
2019-10-01 15:31:55 -07:00
Anders Kaseorg 8d38f0593b run-dev: Disable Tornado response decompression.
Apparently Tornado decompresses gzip responses by default.  Worse, it
fails to adjust the Content-Length header when it does.

https://github.com/tornadoweb/tornado/issues/2743

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-24 15:51:06 -07:00
Tim Abbott a84bb89bdc scripts: Move mobile i18n code out scripts/.
Like other code that is only used in the development environment, this
doesn't belong in scripts/.
2019-09-24 12:57:42 -07:00
Tim Abbott 27b3c1a312 provision: Move install-shellcheck to proper directory.
Scripts in scripts/ should be exclusively code that used in
production, and this isn't.
2019-09-24 12:54:33 -07:00
Anders Kaseorg d368aaad10 requirements: Generate pip.txt from pip.in like the other *.txt files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 13:23:58 -07:00
Anders Kaseorg 92b42573fb requirements: Compile thumbor requirements with pip-tools on Python 2.
The reason that `pip-tools` running on Python 3 didn’t detect the
right requirements for `thumbor` on Python 2 is simply that some of
them are conditional on the Python version.

As for the requirements that had been manually added as a workaround:
`backports-abc` and `singledispatch` are now correctly detected, while
`backports.ssl-match-hostname` was vendored into `urllib3` some time
ago and `certifi` is no longer necessary.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-23 11:56:36 -07:00
Anders Kaseorg fc0554fcb8 emoji: Finish removing leftover code from banned Emoji sets.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-19 12:08:52 -07:00
Tim Abbott 356e68e09e test-backend: Temporarily disable coverage requirement for mattermost.
The mattermost import tool has some temporary coverage regressions.
2019-09-18 12:11:58 -07:00
Anders Kaseorg 9b38bcc2a6 templates: Abstract render_bundle behind a Jinja2 macro.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-17 16:06:33 -07:00
Anders Kaseorg 366dce5d52 confirm_preregistrationuser: Uninline script.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-16 17:23:20 -07:00
Anders Kaseorg 17192ed46b Revert "provision: Give concrete NFS error message on older OSX versions."
This reverts commit 073ecaac66 (#9365).

This exception handler was overly broad in catching all `OSError`s,
and it made debugging harder by hiding the actual exception.
Furthermore, we no longer use NFS (#12963), and we’re now getting
reports of Windows users running into this message.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-09 14:47:51 -07:00
Wyatt Hoodes fc2574fdfd test_mattermost_importer: Add test coverage.
This is a rebase from a recent revert. A test
was failing non-deterministically relating to
`convert_channel_data` and use of an incorrect
`realm_id`.
2019-09-03 12:36:41 -07:00
Anders Kaseorg 177eb2e02e webpack: Move CSS minification to optimization stage.
Otherwise Bootstrap doesn’t get minified, and also the minification
state is incorrectly reflected in the webpack cache.

The Terser plugin is used by default; we need to include it explicitly
to avoid removing it.

Switch from cssnano to clean-css because it’s noticeably faster.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-02 21:58:13 -07:00
Anders Kaseorg e57ede9421 yarn.lock: Share duplicate packages with yarn-deduplicate.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-09-02 19:30:09 -07:00