2021 年的期刊文章 · 第 36 页

共 1542 篇,按发表时间排,新的在前,每页 40 篇。这一页是 2021 到 2021。

按期刊、按主题看 →

第 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 页

JMLR Nonparametric & Semiparametric · 发表于 2021

Generalization Properties of hyper-RKHS and its Applications

Fanghui Liu, Lei Shi, Xiaolin Huang, Jie Yang, Johan A.K. Suykens

摘要

This paper generalizes regularized regression problems in a hyper-reproducing kernel Hilbert space (hyper-RKHS), illustrates its utility for kernel learning and out-of-sample extensions, and proves asymptotic convergence results for the introduced regression models in an approximation theory view. Algorithmically, we consider two regularized regression models with bivariate forms in this space, including kernel ridge regression (KRR) and support vector regression (SVR) endowed with hyper-RKHS, and further combine divide-and-conquer with Nyström approximation for scalability in large sample cases. This framework is general: the underlying kernel is learned from a broad class, and can be positive definite or not, which adapts to various requirements in kernel learning. Theoretically, we study the convergence behavior of regularized regression algorithms in hyper-RKHS and derive the learning rates, which goes beyond the classical analysis on RKHS due to the non-trivial independence of pairwise samples and the characterisation of hyper-RKHS. Experimentally, results on several benchmarks suggest that the employed framework is able to learn a general kernel function form an arbitrary similarity matrix, and thus achieves a satisfactory performance on classification tasks.

JMLR Optimization & Learning Theory · 发表于 2021

Generalization Performance of Multi-pass Stochastic Gradient Descent with Convex Loss Functions

Yunwen Lei, Ting Hu, Ke Tang

摘要

Stochastic gradient descent (SGD) has become the method of choice to tackle large-scale datasets due to its low computational cost and good practical performance. Learning rate analysis, either capacity-independent or capacity-dependent, provides a unifying viewpoint to study the computational and statistical properties of SGD, as well as the implicit regularization by tuning the number of passes. Existing capacity-independent learning rates require a nontrivial bounded subgradient assumption and a smoothness assumption to be optimal. Furthermore, existing capacity-dependent learning rates are only established for the specific least squares loss with a special structure. In this paper, we provide both optimal capacity-independent and capacity-dependent learning rates for SGD with general convex loss functions. Our results require neither bounded subgradient assumptions nor smoothness assumptions, and are stated with high probability. We achieve this improvement by a refined estimate on the norm of SGD iterates based on a careful martingale analysis and concentration inequalities on empirical processes.

Econometrica Econometrics & Economics · 发表于 2021

General Equilibrium Oligopoly and Ownership Structure

José Azar, Xavier Vives

摘要

We develop a tractable general equilibrium framework in which firms are large and have market power with respect to both products and labor, and in which a firm's decisions are affected by its ownership structure. We characterize the Cournot–Walras equilibrium of an economy where each firm maximizes a share‐weighted average of shareholder utilities—rendering the equilibrium independent of price normalization. In a one‐sector economy, if returns to scale are non‐increasing, then an increase in “effective” market concentration (which accounts for common ownership) leads to declines in employment, real wages, and the labor share. Yet when there are multiple sectors, due to an intersectoral pecuniary externality, an increase in common ownership could stimulate the economy when the elasticity of labor supply is high relative to the elasticity of substitution in product markets. We characterize for which ownership structures the monopolistically competitive limit or an oligopolistic one is attained as the number of sectors in the economy increases. When firms have heterogeneous constant returns to scale technologies, we find that an increase in common ownership leads to markets that are more concentrated.

JMLR Bayesian Methods Networks & Graphs High-Dimensional · 发表于 2021

GemBag: Group Estimation of Multiple Bayesian Graphical Models

Xinming Yang, Lingrui Gan, Naveen N. Narisetty, Feng Liang

摘要

In this paper, we propose a novel hierarchical Bayesian model and an efficient estimation method for the problem of joint estimation of multiple graphical models, which have similar but different sparsity structures and signal strength. Our proposed hierarchical Bayesian model is well suited for sharing of sparsity structures, and our procedure, called as GemBag, is shown to enjoy optimal theoretical properties in terms of sup-norm estimation accuracy and correct recovery of the graphical structure even when some of the signals are weak. Although optimization of the posterior distribution required for obtaining our proposed estimator is a non-convex optimization problem, we show that it turns out to be convex in a large constrained space facilitating the use of computationally efficient algorithms. Through extensive simulation studies and an application to a bike sharing data set, we demonstrate that the proposed GemBag procedure has strong empirical performance in comparison with alternative methods.

JMLR Optimization & Learning Theory · 发表于 2021

Gaussian Approximation for Bias Reduction in Q-Learning

Carlo D'Eramo, Andrea Cini, Alessandro Nuara, Matteo Pirotta, Cesare Alippi, Jan Peters, Marcello Restelli

摘要

Temporal-Difference off-policy algorithms are among the building blocks of reinforcement learning (RL). Within this family, Q-Learning is arguably the most famous one, which has been widely studied and extended. The update rule of Q-learning involves the use of the maximum operator to estimate the maximum expected value of the return. However, this estimate is positively biased, and may hinder the learning process, especially in stochastic environments and when function approximation is used. We introduce the Weighted Estimator as an effective solution to mitigate the negative effects of overestimation in Q-Learning. The Weighted Estimator estimates the maximum expected value as a weighted sum of the action values, with the weights being the probabilities that each action value is the maximum. In this work, we study the problem from the statistical perspective of estimating the maximum expected value of a set of random variables and provide bounds to the bias and the variance of the Weighted Estimator, showing its advantages over other estimators present in literature. Then, we derive algorithms to enable the use of the Weighted Estimator, in place of the Maximum Estimator, in online and batch RL, and we introduce a novel algorithm for deep RL. Finally, we empirically evaluate our algorithms in a large set of heterogeneous problems, encompassing discrete and continuous, low and high dimensional, deterministic and stochastic environments. Experimental results show the effectiveness of the Weighted Estimator in controlling the bias of the estimate, resulting in better performance than representative baselines and robust learning w.r.t. a large set of diverse environments.

JMLR Optimization & Learning Theory Bayesian Methods · 发表于 2021

GIBBON: General-purpose Information-Based Bayesian Optimisation

Henry B. Moss, David S. Leslie, Javier Gonzalez, Paul Rayson

摘要

This paper describes a general-purpose extension of max-value entropy search, a popular approach for Bayesian Optimisation (BO). A novel approximation is proposed for the information gain -- an information-theoretic quantity central to solving a range of BO problems, including noisy, multi-fidelity and batch optimisations across both continuous and highly-structured discrete spaces. Previously, these problems have been tackled separately within information-theoretic BO, each requiring a different sophisticated approximation scheme, except for batch BO, for which no computationally-lightweight information-theoretic approach has previously been proposed. GIBBON (General-purpose Information-Based Bayesian OptimisatioN) provides a single principled framework suitable for all the above, out-performing existing approaches whilst incurring substantially lower computational overheads. In addition, GIBBON does not require the problem's search space to be Euclidean and so is the first high-performance yet computationally light-weight acquisition function that supports batch BO over general highly structured input spaces like molecular search and gene design. Moreover, our principled derivation of GIBBON yields a natural interpretation of a popular batch BO heuristic based on determinantal point processes. Finally, we analyse GIBBON across a suite of synthetic benchmark tasks, a molecular search loop, and as part of a challenging batch multi-fidelity framework for problems with controllable experimental noise.

JMLR Latent Variables & Mixtures · 发表于 2021

Further results on latent discourse models and word embeddings

Sammy Khalife, Douglas Gonçalves, Youssef Allouah, Leo Liberti

摘要

We discuss some properties of generative models for word embeddings. Namely, (Arora & Al., 2016) proposed a latent discourse model implying the concentration of the partition function of the word vectors. This concentration phenomenon led to an asymptotic linear relation between the pointwise mutual information (PMI) of pairs of words and the scalar product of their vectors. Here, we first revisit this concentration phenomenon and prove it under slightly weaker assumptions, for a set of random vectors symmetrically distributed around the origin. Second, we empirically evaluate the relation between PMI and scalar products of word vectors satisfying the concentration property. Our empirical results indicate that, in practice, this relation does not hold with arbitrarily small error. This observation is further supported by two theoretical results: (i) the error cannot be exactly zero because the corresponding shifted PMI matrix cannot be positive semidefinite; (ii) under mild assumptions, there exist pairs of words for which the error cannot be close to zero. We deduce that either natural language does not follow the assumptions of the considered generative model, or the current word vector generation methods do not allow the construction of the hypothesized word embeddings.

JMLR Optimization & Learning Theory · 发表于 2021

From Low Probability to High Confidence in Stochastic Convex Optimization

Damek Davis, Dmitriy Drusvyatskiy, Lin Xiao, Junyu Zhang

摘要

Standard results in stochastic convex optimization bound the number of samples that an algorithm needs to generate a point with small function value in expectation. More nuanced high probability guarantees are rare, and typically either rely on light-tail noise assumptions or exhibit worse sample complexity. In this work, we show that a wide class of stochastic optimization algorithms for strongly convex problems can be augmented with high confidence bounds at an overhead cost that is only logarithmic in the confidence level and polylogarithmic in the condition number. The procedure we propose, called proxBoost, is elementary and builds on two well-known ingredients: robust distance estimation and the proximal point method. We discuss consequences for both streaming (online) algorithms and offline algorithms based on empirical risk minimization.

JMLR Spatial & Time Series · 发表于 2021

From Fourier to Koopman: Spectral Methods for Long-term Time Series Prediction

Henning Lange, Steven L. Brunton, J. Nathan Kutz

摘要

We propose spectral methods for long-term forecasting of temporal signals stemming from linear and nonlinear quasi-periodic dynamical systems. For linear signals, we introduce an algorithm with similarities to the Fourier transform but which does not rely on periodicity assumptions, allowing for forecasting given potentially arbitrary sampling intervals. We then extend this algorithm to handle nonlinearities by leveraging Koopman theory. The resulting algorithm performs a spectral decomposition in a nonlinear, data-dependent basis. The optimization objective for both algorithms is highly non-convex. However, expressing the objective in the frequency domain allows us to compute global optima of the error surface in a scalable and efficient manner, partially by exploiting the computational properties of the Fast Fourier Transform. Because of their close relation to Bayesian Spectral Analysis, uncertainty quantification metrics are a natural byproduct of the spectral forecasting methods. We extensively benchmark these algorithms against other leading forecasting methods on a range of synthetic experiments as well as in the context of real-world power systems and fluid flows.

Econometrica Other · 发表于 2021

From Blackwell Dominance in Large Samples to Rényi Divergences and Back Again

Xiaosheng Mu, Luciano Pomatto, Philipp Strack, Omer Tamuz

摘要

We study repeated independent Blackwell experiments; standard examples include drawing multiple samples from a population, or performing a measurement in different locations. In the baseline setting of a binary state of nature, we compare experiments in terms of their informativeness in large samples. Addressing a question due to Blackwell (1951), we show that generically an experiment is more informative than another in large samples if and only if it has higher Rényi divergences. We apply our analysis to the problem of measuring the degree of dissimilarity between distributions by means of divergences. A useful property of Rényi divergences is their additivity with respect to product distributions. Our characterization of Blackwell dominance in large samples implies that every additive divergence that satisfies the data processing inequality is an integral of Rényi divergences.

Biometrics Nonparametric & Semiparametric Clinical Trials · 发表于 2021

Flexible use of copula‐type model for dose‐finding in drug combination clinical trials

Koichi Hashizume, Jun Tshuchida, Takashi Sozu

摘要

Identification of the maximum tolerated dose combination (MTDC) of cancer drugs is an important objective in phase I oncology trials. Numerous dose‐finding designs for drug combination have been proposed over the years. Copula‐type models exhibit distinctive advantages in this task over other models used in existing competitive designs. For example, their application enables the consideration of dose‐limiting toxicities attributable to one of two agents. However, if a particular combination therapy demonstrates extremely synergistic toxicity, copula‐type models are liable to induce biases in toxicity probability estimators due to the associated Fréchet–Hoeffding bounds. Consequently, the dose‐finding performance may be worse than those of other competitive designs. The objective of this study is to improve the performance of dose‐finding designs based on copula‐type models while maintaining their advantageous properties. We propose an extension of the parameter space of the interaction term in copula‐type models. This releases the Fréchet–Hoeffding bounds, making the estimation of toxicity probabilities more flexible. Numerical examples in various scenarios demonstrate that the performance (e.g., the percentage of correct MTDC selection) of the proposed method is better than those exhibited by existing copula‐type models and comparable with those of other competitive designs, irrespective of the existence of extreme synergistic toxicity. The results obtained in this study could motivate the real‐world application of the proposed method in cases requiring the utilization of the properties of copula‐type models.

JMLR Bayesian Methods · 发表于 2021

Flexible Signal Denoising via Flexible Empirical Bayes Shrinkage

Zhengrong Xing, Peter Carbonetto, Matthew Stephens

摘要

Signal denoising—also known as non-parametric regression—is often performed through shrinkage estimation in a transformed (e.g., wavelet) domain; shrinkage in the transformed domain corresponds to smoothing in the original domain. A key question in such applications is how much to shrink, or, equivalently, how much to smooth. Empirical Bayes shrinkage methods provide an attractive solution to this problem; they use the data to estimate a distribution of underlying "effects," hence automatically select an appropriate amount of shrinkage. However, most existing implementations of empirical Bayes shrinkage are less flexible than they could be—both in their assumptions on the underlying distribution of effects, and in their ability to handle heteroskedasticity—which limits their signal denoising applications. Here we address this by adopting a particularly flexible, stable and computationally convenient empirical Bayes shrinkage method and applying it to several signal denoising problems. These applications include smoothing of Poisson data and heteroskedastic Gaussian data. We show through empirical comparisons that the results are competitive with other methods, including both simple thresholding rules and purpose-built empirical Bayes procedures. Our methods are implemented in the R package smashr, "SMoothing by Adaptive SHrinkage in R," available at https://www.github.com/stephenslab/smashr.

JMLR Other · 发表于 2021

First-order Convergence Theory for Weakly-Convex-Weakly-Concave Min-max Problems

Mingrui Liu, Hassan Rafique, Qihang Lin, Tianbao Yang

摘要

In this paper, we consider first-order convergence theory and algorithms for solving a class of non-convex non-concave min-max saddle-point problems, whose objective function is weakly convex in the variables of minimization and weakly concave in the variables of maximization. It has many important applications in machine learning including training Generative Adversarial Nets (GANs). We propose an algorithmic framework motivated by the inexact proximal point method, where the weakly monotone variational inequality (VI) corresponding to the original min-max problem is solved through approximately solving a sequence of strongly monotone VIs constructed by adding a strongly monotone mapping to the original gradient mapping. We prove first-order convergence to a nearly stationary solution of the original min-max problem of the generic algorithmic framework and establish different rates by employing different algorithms for solving each strongly monotone VI. Experiments verify the convergence theory and also demonstrate the effectiveness of the proposed methods on training GANs.

Econometrica Causal Inference · 发表于 2021

Finite‐Sample Optimal Estimation and Inference on Average Treatment Effects Under Unconfoundedness

Timothy B. Armstrong, Michal Kolesár

摘要

We consider estimation and inference on average treatment effects under unconfoundedness conditional on the realizations of the treatment variable and covariates. Given nonparametric smoothness and/or shape restrictions on the conditional mean of the outcome variable, we derive estimators and confidence intervals (CIs) that are optimal in finite samples when the regression errors are normal with known variance. In contrast to conventional CIs, our CIs use a larger critical value that explicitly takes into account the potential bias of the estimator. When the error distribution is unknown, feasible versions of our CIs are valid asymptotically, even when n ‐inference is not possible due to lack of overlap, or low smoothness of the conditional mean. We also derive the minimum smoothness conditions on the conditional mean that are necessary for n ‐inference. When the conditional mean is restricted to be Lipschitz with a large enough bound on the Lipschitz constant, the optimal estimator reduces to a matching estimator with the number of matches set to one. We illustrate our methods in an application to the National Supported Work Demonstration.

JMLR Other · 发表于 2021

Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime

Niladri S. Chatterji, Philip M. Long

摘要

We prove bounds on the population risk of the maximum margin algorithm for two-class linear classification. For linearly separable training data, the maximum margin algorithm has been shown in previous work to be equivalent to a limit of training with logistic loss using gradient descent, as the training error is driven to zero. We analyze this algorithm applied to random data including misclassification noise. Our assumptions on the clean data include the case in which the class-conditional distributions are standard normal distributions. The misclassification noise may be chosen by an adversary, subject to a limit on the fraction of corrupted labels. Our bounds show that, with sufficient over-parameterization, the maximum margin algorithm trained on noisy data can achieve nearly optimal population risk.

JMLR Other · 发表于 2021

Finite Time LTI System Identification

Tuhin Sarkar, Alexander Rakhlin, Munther A. Dahleh

摘要

We address the problem of learning the parameters of a stable linear time invariant (LTI) system with unknown latent space dimension, or order, from a single time--series of noisy input-output data. We focus on learning the best lower order approximation allowed by finite data. Motivated by subspace algorithms in systems theory, where the doubly infinite system Hankel matrix captures both order and good lower order approximations, we construct a Hankel-like matrix from noisy finite data using ordinary least squares. This circumvents the non-convexities that arise in system identification, and allows accurate estimation of the underlying LTI system. Our results rely on careful analysis of self-normalized martingale difference terms that helps bound identification error up to logarithmic factors of the lower bound. We provide a data-dependent scheme for order selection and find an accurate realization of system parameters, corresponding to that order, by an approach that is closely related to the Ho-Kalman subspace algorithm. We demonstrate that the proposed model order selection procedure is not overly conservative, i.e., for the given data length it is not possible to estimate higher order models or find higher order approximations with reasonable accuracy.

JMLR Optimization & Learning Theory · 发表于 2021

Fast Learning for Renewal Optimization in Online Task Scheduling

Michael J. Neely

摘要

This paper considers online optimization of a renewal-reward system. A controller performs a sequence of tasks back-to-back. Each task has a random vector of parameters, called the task type vector, that affects the task processing options and also affects the resulting reward and time duration of the task. The probability distribution for the task type vector is unknown and the controller must learn to make efficient decisions so that time-average reward converges to optimality. Prior work on such renewal optimization problems leaves open the question of optimal convergence time. This paper develops an algorithm with an optimality gap that decays like $O(1/\sqrt{k})$, where $k$ is the number of tasks processed. The same algorithm is shown to have faster $O(\log(k)/k)$ performance when the system satisfies a strong concavity property. The proposed algorithm uses an auxiliary variable that is updated according to a classic Robbins-Monro iteration. It makes online scheduling decisions at the start of each renewal frame based on this variable and the observed task type. A matching converse is obtained for the strongly concave case by constructing an example system for which all algorithms have performance at best $\Omega(\log(k)/k)$. A matching $\Omega(1/\sqrt{k})$ converse is also shown for the general case without strong concavity.

JMLR Optimization & Learning Theory · 发表于 2021

Failures of Model-dependent Generalization Bounds for Least-norm Interpolation

Peter L. Bartlett, Philip M. Long

摘要

We consider bounds on the generalization performance of the least-norm linear regressor, in the over-parameterized regime where it can interpolate the data. We describe a sense in which any generalization bound of a type that is commonly proved in statistical learning theory must sometimes be very loose when applied to analyze the least-norm interpolant. In particular, for a variety of natural joint distributions on training examples, any valid generalization bound that depends only on the output of the learning algorithm, the number of training examples, and the confidence parameter, and that satisfies a mild condition (substantially weaker than monotonicity in sample size), must sometimes be very loose - it can be bounded below by a constant when the true excess risk goes to zero.

JMLR High-Dimensional · 发表于 2021

Factorization Machines with Regularization for Sparse Feature Interactions

Kyohei Atarashi, Satoshi Oyama, Masahito Kurihara

摘要

Factorization machines (FMs) are machine learning predictive models based on second-order feature interactions and FMs with sparse regularization are called sparse FMs. Such regularizations enable feature selection, which selects the most relevant features for accurate prediction, and therefore they can contribute to the improvement of the model accuracy and interpretability. However, because FMs use second-order feature interactions, the selection of features often causes the loss of many relevant feature interactions in the resultant models. In such cases, FMs with regularization specially designed for feature interaction selection trying to achieve interaction-level sparsity may be preferred instead of those just for feature selection trying to achieve feature-level sparsity. In this paper, we present a new regularization scheme for feature interaction selection in FMs. For feature interaction selection, our proposed regularizer makes the feature interaction matrix sparse without a restriction on sparsity patterns imposed by the existing methods. We also describe efficient proximal algorithms for the proposed FMs and how our ideas can be applied or extended to feature selection and other related models such as higher-order FMs and the all-subsets model. The analysis and experimental results on synthetic and real-world datasets show the effectiveness of the proposed methods.

JMLR Causal Inference · 发表于 2021

FLAME: A Fast Large-scale Almost Matching Exactly Approach to Causal Inference

Tianyu Wang, Marco Morucci, M. Usaid Awan, Yameng Liu, Sudeepa Roy, Cynthia Rudin, Alexander Volfovsky

摘要

A classical problem in causal inference is that of matching, where treatment units need to be matched to control units based on covariate information. In this work, we propose a method that computes high quality almost-exact matches for high-dimensional categorical datasets. This method, called FLAME (Fast Large-scale Almost Matching Exactly), learns a distance metric for matching using a hold-out training data set. In order to perform matching efficiently for large datasets, FLAME leverages techniques that are natural for query processing in the area of database management, and two implementations of FLAME are provided: the first uses SQL queries and the second uses bit-vector techniques. The algorithm starts by constructing matches of the highest quality (exact matches on all covariates), and successively eliminates variables in order to match exactly on as many variables as possible, while still maintaining interpretable high-quality matches and balance between treatment and control groups. We leverage these high quality matches to estimate conditional average treatment effects (CATEs). Our experiments show that FLAME scales to huge datasets with millions of observations where existing state-of-the-art methods fail, and that it achieves significantly better performance than other matching methods.

JMLR Econometrics & Economics Meta-Analysis & Data Integration · 发表于 2021

FATE: An Industrial Grade Platform for Collaborative Learning With Data Protection

Yang Liu, Tao Fan, Tianjian Chen, Qian Xu, Qiang Yang

摘要

Collaborative and federated learning has become an emerging solution to many industrial applications where data values from different sites are exploit jointly with privacy protection. We introduce FATE, an industrial-grade project that supports enterprises and institutions to build machine learning models collaboratively at large-scale in a distributed manner. FATE supports a variety of secure computation protocols and machine learning algorithms, and features out-of-box usability with end-to-end building modules and visualization tools. Documentations are available at https://github.com/FederatedAI/FATE. Case studies and other information are available at https://www.fedai.org.

Econometrica Econometrics & Economics · 发表于 2021

Extreme Points and Majorization: Economic Applications

Andreas Kleiner, Benny Moldovanu, Philipp Strack

摘要

We characterize the set of extreme points of monotonic functions that are either majorized by a given function f or themselves majorize f and show that these extreme points play a crucial role in many economic design problems. Our main results show that each extreme point is uniquely characterized by a countable collection of intervals. Outside these intervals the extreme point equals the original function f and inside the function is constant. Further consistency conditions need to be satisfied pinning down the value of an extreme point in each interval where it is constant. We apply these insights to a varied set of economic problems: equivalence and optimality of mechanisms for auctions and (matching) contests, Bayesian persuasion, optimal delegation, and decision making under uncertainty.

JMLR Other · 发表于 2021

Explaining by Removing: A Unified Framework for Model Explanation

Ian Covert, Scott Lundberg, Su-In Lee

摘要

Researchers have proposed a wide variety of model explanation approaches, but it remains unclear how most methods are related or when one method is preferable to another. We describe a new unified class of methods, removal-based explanations, that are based on the principle of simulating feature removal to quantify each feature's influence. These methods vary in several respects, so we develop a framework that characterizes each method along three dimensions: 1) how the method removes features, 2) what model behavior the method explains, and 3) how the method summarizes each feature's influence. Our framework unifies 26 existing methods, including several of the most widely used approaches: SHAP, LIME, Meaningful Perturbations, and permutation tests. This newly understood class of explanation methods has rich connections that we examine using tools that have been largely overlooked by the explainability literature. To anchor removal-based explanations in cognitive psychology, we show that feature removal is a simple application of subtractive counterfactual reasoning. Ideas from cooperative game theory shed light on the relationships and trade-offs among different methods, and we derive conditions under which all removal-based explanations have information-theoretic interpretations. Through this analysis, we develop a unified framework that helps practitioners better understand model explanation tools, and that offers a strong theoretical foundation upon which future explainability research can build.

JMLR Networks & Graphs Machine Learning · 发表于 2021

Explaining Explanations: Axiomatic Feature Interactions for Deep Networks

Joseph D. Janizek, Pascal Sturmfels, Su-In Lee

摘要

Recent work has shown great promise in explaining neural network behavior. In particular, feature attribution methods explain the features that are important to a model's prediction on a given input. However, for many tasks, simply identifying significant features may be insufficient for understanding model behavior. The interactions between features within the model may better explain not only the model, but why certain features outrank others in importance. In this work, we present Integrated Hessians, an extension of Integrated Gradients that explains pairwise feature interactions in neural networks. Integrated Hessians overcomes several theoretical limitations of previous methods, and unlike them, is not limited to a specific architecture or class of neural network. Additionally, we find that our method is faster than existing methods when the number of features is large, and outperforms previous methods on existing quantitative benchmarks.

JMLR Other · 发表于 2021

Expanding Boundaries of Gap Safe Screening

Cassio F. Dantas, Emmanuel Soubies, Cédric Févotte

摘要

Sparse optimization problems are ubiquitous in many fields such as statistics, signal/image processing and machine learning. This has led to the birth of many iterative algorithms to solve them. A powerful strategy to boost the performance of these algorithms is known as safe screening: it allows the early identification of zero coordinates in the solution, which can then be eliminated to reduce the problem's size and accelerate convergence. In this work, we extend the existing Gap Safe screening framework by relaxing the global strong-concavity assumption on the dual cost function. Instead, we exploit local regularity properties, that is, strong concavity on well-chosen subsets of the domain. The non-negativity constraint is also integrated to the existing framework. Besides making safe screening possible to a broader class of functions that includes $\beta$-divergences (e.g., the Kullback-Leibler divergence), the proposed approach also improves upon the existing Gap Safe screening rules on previously applicable cases (e.g., logistic regression). The proposed general framework is exemplified by some notable particular cases: logistic function, $\beta=1.5$ and Kullback-Leibler divergences. Finally, we showcase the effectiveness of the proposed screening rules with different solvers (coordinate descent, multiplicative-update and proximal gradient algorithms) and different datasets (binary classification, hyperspectral and count data).

Econometrica Econometrics & Economics · 发表于 2021

Exchange Design and Efficiency

Marzena Rostek, Ji Hee Yoon

摘要

Most assets clear independently rather than jointly. This paper presents a model based on the uniform‐price double auction which accommodates arbitrary restrictions on market clearing, including independent clearing across assets (allowed when demand for each asset is contingent only on the price of that asset) and joint market clearing for all assets (required when demand for each asset is contingent on the prices of all assets). Additional trading protocols for traded assets—neutral when the market clears jointly—are generally not redundant innovations, even if all traders participate in all protocols. Multiple trading protocols that clear independently can be designed to be at least as efficient as joint market clearing for all assets. The change in price impact brought by independence in market clearing can overcome the loss of information, and enhance diversification and risk sharing. Except when the market is competitive, market characteristics should guide innovation in trading technology.

JMLR Optimization & Learning Theory · 发表于 2021

Exact Asymptotics for Linear Quadratic Adaptive Control

Feicheng Wang, Lucas Janson

摘要

Recent progress in reinforcement learning has led to remarkable performance in a range of applications, but its deployment in high-stakes settings remains quite rare. One reason is a limited understanding of the behavior of reinforcement algorithms, both in terms of their regret and their ability to learn the underlying system dynamics---existing work is focused almost exclusively on characterizing rates, with little attention paid to the constants multiplying those rates that can be critically important in practice. To start to address this challenge, we study perhaps the simplest non-bandit reinforcement learning problem: linear quadratic adaptive control (LQAC). By carefully combining recent finite-sample performance bounds for the LQAC problem with a particular (less-recent) martingale central limit theorem, we are able to derive asymptotically exact expressions for the regret, estimation error, and prediction error of a rate-optimal stepwise-updating LQAC algorithm. In simulations on both stable and unstable systems, we find that our asymptotic theory also describes the algorithm's finite-sample behavior remarkably well.

JMLR Causal Inference Optimization & Learning Theory · 发表于 2021

Estimation and Optimization of Composite Outcomes

Daniel J. Luckett, Eric B. Laber, Siyeon Kim, Michael R. Kosorok

摘要

There is tremendous interest in precision medicine as a means to improve patient outcomes by tailoring treatment to individual characteristics. An individualized treatment rule formalizes precision medicine as a map from patient information to a recommended treatment. A treatment rule is defined to be optimal if it maximizes the mean of a scalar outcome in a population of interest, e.g., symptom reduction. However, clinical and intervention scientists often seek to balance multiple and possibly competing outcomes, e.g., symptom reduction and the risk of an adverse event. One approach to precision medicine in this setting is to elicit a composite outcome which balances all competing outcomes; unfortunately, eliciting a composite outcome directly from patients is difficult without a high-quality instrument, and an expert-derived composite outcome may not account for heterogeneity in patient preferences. We propose a new paradigm for the study of precision medicine using observational data that relies solely on the assumption that clinicians are approximately (i.e., imperfectly) making decisions to maximize individual patient utility. Estimated composite outcomes are subsequently used to construct an estimator of an individualized treatment rule which maximizes the mean of patient-specific composite outcomes. The estimated composite outcomes and estimated optimal individualized treatment rule provide new insights into patient preference heterogeneity, clinician behavior, and the value of precision medicine in a given domain. We derive inference procedures for the proposed estimators under mild conditions and demonstrate their finite sample performance through a suite of simulation experiments and an illustrative application to data from a study of bipolar depression.

JMLR High-Dimensional Testing & Uncertainty · 发表于 2021

Estimation and Inference for High Dimensional Generalized Linear Models: A Splitting and Smoothing Approach

Zhe Fei, Yi Li

摘要

The focus of modern biomedical studies has gradually shifted to explanation and estimation of joint effects of high dimensional predictors on disease risks. Quantifying uncertainty in these estimates may provide valuable insight into prevention strategies or treatment decisions for both patients and physicians. High dimensional inference, including confidence intervals and hypothesis testing, has sparked much interest. While much work has been done in the linear regression setting, there is lack of literature on inference for high dimensional generalized linear models. We propose a novel and computationally feasible method, which accommodates a variety of outcome types, including normal, binomial, and Poisson data. We use a “splitting and smoothing” approach, which splits samples into two parts, performs variable selection using one part and conducts partial regression with the other part. Averaging the estimates over multiple random splits, we obtain the smoothed estimates, which are numerically stable. We show that the estimates are consistent, asymptotically normal, and construct confidence intervals with proper coverage probabilities for all predictors. We examine the finite sample performance of our method by comparing it with the existing methods and applying it to analyze a lung cancer cohort study.

JMLR High-Dimensional · 发表于 2021

Estimating the Lasso's Effective Noise

Johannes Lederer, Michael Vogt

摘要

Much of the theory for the lasso in the linear model $Y = \boldsymbol{X} \beta^* + \varepsilon$ hinges on the quantity $2\| \boldsymbol{X}^\top \varepsilon \|_\infty / n$, which we call the lasso's effective noise. Among other things, the effective noise plays an important role in finite-sample bounds for the lasso, the calibration of the lasso's tuning parameter, and inference on the parameter vector $\beta^*$. In this paper, we develop a bootstrap-based estimator of the quantiles of the effective noise. The estimator is fully data-driven, that is, does not require any additional tuning parameters. We equip our estimator with finite-sample guarantees and apply it to tuning parameter calibration for the lasso and to high-dimensional inference on the parameter vector $\beta^*$.

JMLR Networks & Graphs Econometrics & Economics · 发表于 2021

Estimating Uncertainty Intervals from Collaborating Networks

Tianhui Zhou, Yitong Li, Yuan Wu, David Carlson

摘要

Effective decision making requires understanding the uncertainty inherent in a prediction. In regression, this uncertainty can be estimated by a variety of methods; however, many of these methods are laborious to tune, generate overconfident uncertainty intervals, or lack sharpness (give imprecise intervals). We address these challenges by proposing a novel method to capture predictive distributions in regression by defining two neural networks with two distinct loss functions. Specifically, one network approximates the cumulative distribution function, and the second network approximates its inverse. We refer to this method as Collaborating Networks (CN). Theoretical analysis demonstrates that a fixed point of the optimization is at the idealized solution, and that the method is asymptotically consistent to the ground truth distribution. Empirically, learning is straightforward and robust. We benchmark CN against several common approaches on two synthetic and six real-world datasets, including forecasting A1c values in diabetic patients from electronic health records, where uncertainty is critical. In the synthetic data, the proposed approach essentially matches ground truth. In the real-world datasets, CN improves results on many performance metrics, including log-likelihood estimates, mean absolute errors, coverage estimates, and prediction interval widths.

Econometrica Nonparametric & Semiparametric Missing Data & Measurement Error · 发表于 2021

Errors in the Dependent Variable of Quantile Regression Models

Jerry Hausman, Haoyang Liu, Ye Luo, Christopher Palmer

摘要

We study the consequences of measurement error in the dependent variable of random‐coefficients models, focusing on the particular case of quantile regression. The popular quantile regression estimator of Koenker and Bassett (1978) is biased if there is an additive error term. Approaching this problem as an errors‐in‐variables problem where the dependent variable suffers from classical measurement error, we present a sieve maximum likelihood approach that is robust to left‐hand‐side measurement error. After providing sufficient conditions for identification, we demonstrate that when the number of knots in the quantile grid is chosen to grow at an adequate speed, the sieve‐maximum‐likelihood estimator is consistent and asymptotically normal, permitting inference via bootstrapping. Monte Carlo evidence verifies our method outperforms quantile regression in mean bias and MSE. Finally, we illustrate our estimator with an application to the returns to education highlighting changes over time in the returns to education that have previously been masked by measurement‐error bias.

Econometrica Other · 发表于 2021

Equitable Voting Rules

Laurent Bartholdi, Wade Hann-Caruthers, Maya Josyula, Omer Tamuz, Leeat Yariv

摘要

May's theorem (1952), a celebrated result in social choice, provides the foundation for majority rule. May's crucial assumption of symmetry, often thought of as a procedural equity requirement, is violated by many choice procedures that grant voters identical roles. We show that a weakening of May's symmetry assumption allows for a far richer set of rules that still treat voters equally. We show that such rules can have minimal winning coalitions comprising a vanishing fraction of the population, but not less than the square root of the population size. Methodologically, we introduce techniques from group theory and illustrate their usefulness for the analysis of social choice questions.

Econometrica Econometrics & Economics · 发表于 2021

Equilibrium Allocations Under Alternative Waitlist Designs: Evidence From Deceased Donor Kidneys

Nikhil Agarwal, Itai Ashlagi, Michael A. Rees, Paulo Somaini, Daniel Waldinger

摘要

Waitlists are often used to ration scarce resources, but the trade‐offs in designing these mechanisms depend on agents' preferences. We study equilibrium allocations under alternative designs for the deceased donor kidney waitlist. We model the decision to accept an organ or wait for a preferable one as an optimal stopping problem and estimate preferences using administrative data from the New York City area. Our estimates show that while some kidney types are desirable for all patients, there is substantial match‐specific heterogeneity in values. We then develop methods to evaluate alternative mechanisms, comparing their effects on patient welfare to an equivalent change in donor supply. Past reforms to the kidney waitlist primarily resulted in redistribution, with similar welfare and organ discard rates to the benchmark first‐come, first‐served mechanism. These mechanisms and other commonly studied theoretical benchmarks remain far from optimal. We design a mechanism that increases patient welfare by the equivalent of an 18.2% increase in donor supply.

JMLR Nonparametric & Semiparametric · 发表于 2021

Entangled Kernels - Beyond Separability

Riikka Huusari, Hachem Kadri

摘要

We consider the problem of operator-valued kernel learning and investigate the possibility of going beyond the well-known separable kernels. Borrowing tools and concepts from the field of quantum computing, such as partial trace and entanglement, we propose a new view on operator-valued kernels and define a general family of kernels that encompasses previously known operator-valued kernels, including separable and transformable kernels. Within this framework, we introduce another novel class of operator-valued kernels called entangled kernels that are not separable. We propose an efficient two-step algorithm for this framework, where the entangled kernel is learned based on a novel extension of kernel alignment to operator-valued kernels. We illustrate our algorithm with an application to supervised dimensionality reduction, and demonstrate its effectiveness with both artificial and real data for multi-output regression.

JMLR Bayesian Methods Optimization & Learning Theory High-Dimensional · 发表于 2021

Empirical Bayes Matrix Factorization

Wei Wang, Matthew Stephens

摘要

Matrix factorization methods, which include Factor analysis (FA) and Principal Components Analysis (PCA), are widely used for inferring and summarizing structure in multivariate data. Many such methods use a penalty or prior distribution to achieve sparse representations (“Sparse FA/PCA"), and a key question is how much sparsity to induce. Here we introduce a general Empirical Bayes approach to matrix factorization (EBMF), whose key feature is that it estimates the appropriate amount of sparsity by estimating prior distributions from the observed data. The approach is very flexible: it allows for a wide range of different prior families and allows that each component of the matrix factorization may exhibit a different amount of sparsity. The key to this flexibility is the use of a variational approximation, which we show effectively reduces fitting the EBMF model to solving a simpler problem, the so-called “normal means" problem. We demonstrate the benefits of EBMF with sparse priors through both numerical comparisons with competing methods and through analysis of data from the GTEx (Genotype Tissue Expression) project on genetic associations across 44 human tissues. In numerical comparisons EBMF often provides more accurate inferences than other methods. In the GTEx data, EBMF identifies interpretable structure that agrees with known relationships among human tissues. Software implementing our approach is available at https://github.com/stephenslab/flashr.

Biometrics Missing Data & Measurement Error Clinical Trials Nonparametric & Semiparametric · 发表于 2021

Efficient odds ratio estimation under two‐phase sampling using error‐prone data from a multi‐national HIV research cohort

Sarah C. Lotspeich, Bryan E. Shepherd, Gustavo G. C. Amorim, Pamela A. Shaw, Ran Tao

摘要

Persons living with HIV engage in routine clinical care, generating large amounts of data in observational HIV cohorts. These data are often error‐prone, and directly using them in biomedical research could bias estimation and give misleading results. A cost‐effective solution is the two‐phase design, under which the error‐prone variables are observed for all patients during Phase I, and that information is used to select patients for data auditing during Phase II. For example, the Caribbean, Central, and South America network for HIV epidemiology (CCASAnet) selected a random sample from each site for data auditing. Herein, we consider efficient odds ratio estimation with partially audited, error‐prone data. We propose a semiparametric approach that uses all information from both phases and accommodates a number of error mechanisms. We allow both the outcome and covariates to be error‐prone and these errors to be correlated, and selection of the Phase II sample can depend on Phase I data in an arbitrary manner. We devise a computationally efficient, numerically stable EM algorithm to obtain estimators that are consistent, asymptotically normal, and asymptotically efficient. We demonstrate the advantages of the proposed methods over existing ones through extensive simulations. Finally, we provide applications to the CCASAnet cohort.

JMLR Networks & Graphs · 发表于 2021

Edge Sampling Using Local Network Information

Can M. Le

摘要

Edge sampling is an important topic in network analysis. It provides a natural way to reduce network size while retaining desired features of the original network. Sampling methods that only use local information are common in practice as they do not require access to the entire network and can be easily parallelized. Despite promising empirical performances, most of these methods are derived from heuristic considerations and lack theoretical justification. In this paper, we study a simple and efficient edge sampling method that uses local network information. We show that when the local connectivity is sufficiently strong, the sampled network satisfies a strong spectral property. We measure the strength of local connectivity by a global parameter and relate it to more common network statistics such as the clustering coefficient and network curvature. Based on this result, we also provide sufficient conditions under which random networks and hypergraphs can be efficiently sampled.

Econometrica High-Dimensional Econometrics & Economics · 发表于 2021

Economic Predictions With Big Data: The Illusion of Sparsity

Domenico Giannone, Michele Lenza, Giorgio E. Primiceri

摘要

We compare sparse and dense representations of predictive models in macroeconomics, microeconomics, and finance. To deal with a large number of possible predictors, we specify a prior that allows for both variable selection and shrinkage. The posterior distribution does not typically concentrate on a single sparse model, but on a wide set of models that often include many predictors.

Econometrica Econometrics & Economics · 发表于 2021

Econometrics for Decision Making: Building Foundations Sketched by Haavelmo and Wald

Charles F. Manski

摘要

Haavelmo (1944) proposed a probabilistic structure for econometric modeling, aiming to make econometrics useful for decision making. His fundamental contribution has become thoroughly embedded in econometric research, yet it could not answer all the deep issues that the author raised. Notably, Haavelmo struggled to formalize the implications for decision making of the fact that models can at most approximate actuality. In the same period, Wald (1939, 1945) initiated his own seminal development of statistical decision theory. Haavelmo favorably cited Wald, but econometrics did not embrace statistical decision theory. Instead, it focused on study of identification, estimation, and statistical inference. This paper proposes use of statistical decision theory to evaluate the performance of models in decision making. I consider the common practice of as‐if optimization : specification of a model, point estimation of its parameters, and use of the point estimate to make a decision that would be optimal if the estimate were accurate. A central theme is that one should evaluate as‐if optimization or any other model‐based decision rule by its performance across the state space, listing all states of nature that one believes feasible, not across the model space. I apply the theme to prediction and treatment choice. Statistical decision theory is conceptually simple, but application is often challenging. Advancing computation is the primary task to complete the foundations sketched by Haavelmo and Wald.

第 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 页